[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:45:20 dist-test-slave-5bvn krb5kdc[1653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:45:20 dist-test-slave-5bvn krb5kdc[1653](info): set up 2 sockets
Jun 20 21:45:20 dist-test-slave-5bvn krb5kdc[1653](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 21:45:21.138406 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:45:21 dist-test-slave-5bvn krb5kdc[1653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:45:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:45:21.261271 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:41197
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:41197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:21.367540 1666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:21.367835 1666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:21.367914 1666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:21.371996 1666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:21.372071 1666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:21.372105 1666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:21.372143 1666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:21.372179 1666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:45:21.377244 1666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:41197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:41197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:21.378296 1666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:21.379302 1666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:21.387744 1671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:21.387766 1674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:21.387804 1666 server_base.cc:1034] running on GCE node
W20260620 21:45:21.387753 1672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:21.388512 1666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:21.389616 1666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:21.390777 1666 hybrid_clock.cc:648] HybridClock initialized: now 1781991921390717 us; error 70 us; skew 500 ppm
Jun 20 21:45:21 dist-test-slave-5bvn krb5kdc[1653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:21.393383 1666 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:45:21.394316 1666 webserver.cc:460] Webserver started at http://127.1.155.190:45529/ using document root <none> and password file <none>
I20260620 21:45:21.394879 1666 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:21.394969 1666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:21.395197 1666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:21.397219 1666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "ba20ff8dd4ea41e786345305d180a0f0"
format_stamp: "Formatted at 2026-06-20 21:45:21 on dist-test-slave-5bvn"
server_key: "74b5dbb0caf4489f53c3a9a057c0c8a5"
server_key_iv: "aa4e3b05a4df0c29211b6b3b229eb528"
server_key_version: "encryptionkey@0"
I20260620 21:45:21.397804 1666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "ba20ff8dd4ea41e786345305d180a0f0"
format_stamp: "Formatted at 2026-06-20 21:45:21 on dist-test-slave-5bvn"
server_key: "74b5dbb0caf4489f53c3a9a057c0c8a5"
server_key_iv: "aa4e3b05a4df0c29211b6b3b229eb528"
server_key_version: "encryptionkey@0"
I20260620 21:45:21.401422 1666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:45:21.403853 1681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:21.405045 1666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:45:21.405194 1666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "ba20ff8dd4ea41e786345305d180a0f0"
format_stamp: "Formatted at 2026-06-20 21:45:21 on dist-test-slave-5bvn"
server_key: "74b5dbb0caf4489f53c3a9a057c0c8a5"
server_key_iv: "aa4e3b05a4df0c29211b6b3b229eb528"
server_key_version: "encryptionkey@0"
I20260620 21:45:21.405304 1666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:21.433713 1666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:21.436887 1666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:21.437108 1666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:21.444340 1666 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:41197
I20260620 21:45:21.444386 1733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:41197 every 8 connection(s)
I20260620 21:45:21.445331 1666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:21.447257 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 1666
I20260620 21:45:21.447424 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:21.448311 1734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:21.448735 1646 external_mini_cluster.cc:1428] Setting key 5e9ff19ae0de62b579e9838a7deae28f
I20260620 21:45:21.454404 1734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0: Bootstrap starting.
I20260620 21:45:21.456492 1734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:21.457302 1734 log.cc:824] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:21.459064 1734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0: No bootstrap required, opened a new log
I20260620 21:45:21.462409 1734 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba20ff8dd4ea41e786345305d180a0f0" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 41197 } }
I20260620 21:45:21.462641 1734 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:21.462719 1734 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba20ff8dd4ea41e786345305d180a0f0, State: Initialized, Role: FOLLOWER
I20260620 21:45:21.463212 1734 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba20ff8dd4ea41e786345305d180a0f0" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 41197 } }
I20260620 21:45:21.463389 1734 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:21.463496 1734 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:21.463593 1734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:21.464450 1734 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba20ff8dd4ea41e786345305d180a0f0" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 41197 } }
I20260620 21:45:21.464831 1734 leader_election.cc:302] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba20ff8dd4ea41e786345305d180a0f0; no voters:
I20260620 21:45:21.465194 1734 leader_election.cc:288] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:21.465279 1739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [term 1 FOLLOWER]: Leader election won for term 1
Jun 20 21:45:21 dist-test-slave-5bvn krb5kdc[1653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:21.465562 1739 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [term 1 LEADER]: Becoming Leader. State: Replica: ba20ff8dd4ea41e786345305d180a0f0, State: Running, Role: LEADER
I20260620 21:45:21.465907 1739 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba20ff8dd4ea41e786345305d180a0f0" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 41197 } }
I20260620 21:45:21.466548 1734 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:21.467976 1741 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba20ff8dd4ea41e786345305d180a0f0. Latest consensus state: current_term: 1 leader_uuid: "ba20ff8dd4ea41e786345305d180a0f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba20ff8dd4ea41e786345305d180a0f0" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 41197 } } }
I20260620 21:45:21.468093 1741 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:21.468436 1740 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba20ff8dd4ea41e786345305d180a0f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba20ff8dd4ea41e786345305d180a0f0" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 41197 } } }
I20260620 21:45:21.468564 1740 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0 [sys.catalog]: This master's current role is: LEADER
W20260620 21:45:21.471566 1754 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:45:21.471637 1754 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:45:21.471696 1748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:21.472527 1748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:21.472895 1736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:21.451818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.1.155.190:41197)
0620 21:45:21.452216 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:21.452231 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:21.453273 (+ 1042us) server_negotiation.cc:407] Connection header received
0620 21:45:21.456337 (+ 3064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:21.456352 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:21.456694 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:21.457061 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:21.459168 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:21.460141 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:21.461304 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:21.461764 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:21.461989 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:21.466438 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:21.466445 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:21.466460 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:21.466490 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:21.468689 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:21.469360 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:21.469366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:21.469371 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:21.469487 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:21.469829 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:21.469852 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:21.469853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:21.470304 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:21.470488 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:21.471287 (+ 799us) server_negotiation.cc:299] Negotiation successful
0620 21:45:21.471559 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":134,"threads_started":1}
I20260620 21:45:21.477556 1748 catalog_manager.cc:1347] Generated new cluster ID: cb6ef57b500a4a76abde83450ca500cb
I20260620 21:45:21.477629 1748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:21.489758 1748 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:21.490757 1748 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:21.495914 1748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba20ff8dd4ea41e786345305d180a0f0: Generated new TSK 0
I20260620 21:45:21.496536 1748 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:45:21.545377 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:41197
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:21.658741 1760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:21.659044 1760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:21.659118 1760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:21.663344 1760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:21.663427 1760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:21.663516 1760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:45:21.668946 1760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:41197
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:21.670046 1760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:21.671033 1760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:21.680547 1765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:21.680552 1766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:21.680601 1768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:21.680929 1760 server_base.cc:1034] running on GCE node
I20260620 21:45:21.681330 1760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:21.682003 1760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:21.683183 1760 hybrid_clock.cc:648] HybridClock initialized: now 1781991921683143 us; error 47 us; skew 500 ppm
Jun 20 21:45:21 dist-test-slave-5bvn krb5kdc[1653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:21.686043 1760 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:45:21.686903 1760 webserver.cc:460] Webserver started at http://127.1.155.129:46271/ using document root <none> and password file <none>
I20260620 21:45:21.687482 1760 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:21.687577 1760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:21.687806 1760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:21.689805 1760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "de320878281a476b8a087600b33b9b25"
format_stamp: "Formatted at 2026-06-20 21:45:21 on dist-test-slave-5bvn"
server_key: "2d1716994daa7a49c8ca048ff54fd7d6"
server_key_iv: "241e0ee1c5fdf402c0ad838c6227e166"
server_key_version: "encryptionkey@0"
I20260620 21:45:21.690423 1760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "de320878281a476b8a087600b33b9b25"
format_stamp: "Formatted at 2026-06-20 21:45:21 on dist-test-slave-5bvn"
server_key: "2d1716994daa7a49c8ca048ff54fd7d6"
server_key_iv: "241e0ee1c5fdf402c0ad838c6227e166"
server_key_version: "encryptionkey@0"
I20260620 21:45:21.694010 1760 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:45:21.696363 1775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:21.697515 1760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:45:21.697649 1760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "de320878281a476b8a087600b33b9b25"
format_stamp: "Formatted at 2026-06-20 21:45:21 on dist-test-slave-5bvn"
server_key: "2d1716994daa7a49c8ca048ff54fd7d6"
server_key_iv: "241e0ee1c5fdf402c0ad838c6227e166"
server_key_version: "encryptionkey@0"
I20260620 21:45:21.697762 1760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:21.711967 1760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:21.714860 1760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:21.715047 1760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:21.715616 1760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:21.716434 1760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:21.716512 1760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:21.716584 1760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:21.716634 1760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:21.726130 1760 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:41843
I20260620 21:45:21.726152 1888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:41843 every 8 connection(s)
I20260620 21:45:21.726992 1760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:45:21.732525 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 1760
I20260620 21:45:21.732631 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:45:21.732880 1646 external_mini_cluster.cc:1428] Setting key 073d3cb367805063e2e02ea5df65fdfc
Jun 20 21:45:21 dist-test-slave-5bvn krb5kdc[1653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:21.741586 1736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:21.728741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:51019 (local address 127.1.155.190:41197)
0620 21:45:21.728910 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:21.728913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:21.729682 (+ 769us) server_negotiation.cc:407] Connection header received
0620 21:45:21.730981 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:21.730983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:21.731029 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:21.731105 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:21.732362 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:21.733158 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:21.734606 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:21.734989 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:21.735123 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:21.737890 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:21.737898 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:21.737900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:21.737923 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:21.739460 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:21.740049 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:21.740054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:21.740056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:21.740144 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:21.740490 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:21.740495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:21.740497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:21.740685 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:21.740792 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:21.741288 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 21:45:21.741410 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
I20260620 21:45:21.742491 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:21.729063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:41197 (local address 127.1.155.129:51019)
0620 21:45:21.729513 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:21.729558 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:21.730798 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:21.731349 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:21.731356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:21.731706 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:21.732179 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:21.732193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:21.733285 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:21.733288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:21.734486 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:21.734498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:21.735151 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:21.735154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:21.735291 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:21.735954 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:21.735981 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:21.737695 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:21.739650 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:21.739657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:21.739672 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:21.739939 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:21.740246 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:21.740249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:21.740251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:21.740392 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:21.740801 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:21.740810 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:21.741060 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 21:45:21.741250 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 21:45:21.741565 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":120,"threads_started":1}
I20260620 21:45:21.743794 1889 heartbeater.cc:347] Connected to a master server at 127.1.155.190:41197
I20260620 21:45:21.744113 1889 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:21.744652 1889 heartbeater.cc:511] Master 127.1.155.190:41197 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
I20260620 21:45:21.746459 1698 ts_manager.cc:194] Registered new tserver with Master: de320878281a476b8a087600b33b9b25 (127.1.155.129:41843)
I20260620 21:45:21.747993 1698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:51019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:45:21.760598 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:41197
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:21.869218 1894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:21.869490 1894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:21.869566 1894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:21.873646 1894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:21.873728 1894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:21.873816 1894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:45:21.878921 1894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:41197
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:21.879906 1894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:21.880872 1894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:21.890347 1900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:21.890347 1902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:21.890422 1894 server_base.cc:1034] running on GCE node
W20260620 21:45:21.890347 1899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:21.890967 1894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:21.891532 1894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:21.892714 1894 hybrid_clock.cc:648] HybridClock initialized: now 1781991921892701 us; error 23 us; skew 500 ppm
Jun 20 21:45:21 dist-test-slave-5bvn krb5kdc[1653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:21.895457 1894 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:45:21.896341 1894 webserver.cc:460] Webserver started at http://127.1.155.130:37165/ using document root <none> and password file <none>
I20260620 21:45:21.896878 1894 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:21.896966 1894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:21.897212 1894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:21.899247 1894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "7af0f922acfd452f8331e521606980c6"
format_stamp: "Formatted at 2026-06-20 21:45:21 on dist-test-slave-5bvn"
server_key: "a6205db3bcd685d89a7d9414b7c04a6f"
server_key_iv: "eb57fe1566d978ecaee1b31265df49fb"
server_key_version: "encryptionkey@0"
I20260620 21:45:21.899798 1894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "7af0f922acfd452f8331e521606980c6"
format_stamp: "Formatted at 2026-06-20 21:45:21 on dist-test-slave-5bvn"
server_key: "a6205db3bcd685d89a7d9414b7c04a6f"
server_key_iv: "eb57fe1566d978ecaee1b31265df49fb"
server_key_version: "encryptionkey@0"
I20260620 21:45:21.903745 1894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 21:45:21.906404 1909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:21.907698 1894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:45:21.907848 1894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "7af0f922acfd452f8331e521606980c6"
format_stamp: "Formatted at 2026-06-20 21:45:21 on dist-test-slave-5bvn"
server_key: "a6205db3bcd685d89a7d9414b7c04a6f"
server_key_iv: "eb57fe1566d978ecaee1b31265df49fb"
server_key_version: "encryptionkey@0"
I20260620 21:45:21.907974 1894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:21.928074 1894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:21.931092 1894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:21.931299 1894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:21.931869 1894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:21.932667 1894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:21.932711 1894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:21.932780 1894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:21.932813 1894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:21.941620 1894 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:39937
I20260620 21:45:21.941684 2022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:39937 every 8 connection(s)
I20260620 21:45:21.942559 1894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:45:21.946681 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 1894
I20260620 21:45:21.946825 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:45:21.947103 1646 external_mini_cluster.cc:1428] Setting key 8c0a779996fcaff2b057be3e9dea6045
Jun 20 21:45:21 dist-test-slave-5bvn krb5kdc[1653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
I20260620 21:45:21.958304 1736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:21.944469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:40891 (local address 127.1.155.190:41197)
0620 21:45:21.944605 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:21.944608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:21.945877 (+ 1269us) server_negotiation.cc:407] Connection header received
0620 21:45:21.947365 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:21.947368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:21.947426 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:21.947536 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:21.948864 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:21.949390 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:21.951245 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:21.951462 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:21.951552 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:21.954408 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:21.954414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:21.954417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:21.954453 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:21.956071 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:21.956563 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:21.956570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:21.956574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:21.956670 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:21.956985 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:21.956990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:21.956995 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:21.957263 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:21.957378 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:21.957941 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 21:45:21.958112 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:45:21.959115 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:21.944836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:41197 (local address 127.1.155.130:40891)
0620 21:45:21.945695 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:21.945741 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:21.947119 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:21.947724 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:21.947731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:21.948194 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:21.948725 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:21.948739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:21.949513 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:21.949517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:21.951097 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:21.951112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:21.951616 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:21.951623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:21.951850 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:21.952482 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:21.952509 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:21.954232 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:21.956185 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:21.956190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:21.956202 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:21.956439 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:21.956762 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:21.956764 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:21.956765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:21.956885 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:21.957368 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:21.957373 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:21.957605 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 21:45:21.957801 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 21:45:21.958171 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":512,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
I20260620 21:45:21.960361 2023 heartbeater.cc:347] Connected to a master server at 127.1.155.190:41197
I20260620 21:45:21.960609 2023 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:21.961108 2023 heartbeater.cc:511] Master 127.1.155.190:41197 requested a full tablet report, sending...
I20260620 21:45:21.962198 1698 ts_manager.cc:194] Registered new tserver with Master: 7af0f922acfd452f8331e521606980c6 (127.1.155.130:39937)
I20260620 21:45:21.962770 1698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:40891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:45:21.976975 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:41197
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:22.086215 2028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:22.086465 2028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:22.086508 2028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:22.090464 2028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:22.090516 2028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:22.090615 2028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:45:22.095885 2028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:41197
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:22.096951 2028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:22.097965 2028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:45:22.107645 2028 server_base.cc:1034] running on GCE node
W20260620 21:45:22.107657 2034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:22.107564 2033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:22.107564 2036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:22.108307 2028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:22.108985 2028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:22.110174 2028 hybrid_clock.cc:648] HybridClock initialized: now 1781991922110147 us; error 52 us; skew 500 ppm
Jun 20 21:45:22 dist-test-slave-5bvn krb5kdc[1653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991922, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:22.113030 2028 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:45:22.113981 2028 webserver.cc:460] Webserver started at http://127.1.155.131:35779/ using document root <none> and password file <none>
I20260620 21:45:22.114503 2028 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:22.114562 2028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:22.114720 2028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:22.116688 2028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "7cc758ec7e3d487ba586d3a0fa7e501b"
format_stamp: "Formatted at 2026-06-20 21:45:22 on dist-test-slave-5bvn"
server_key: "064fbf2e3917280f0087009ba5a15e27"
server_key_iv: "44c35f06b2a81b3ecb1ea8469e89740c"
server_key_version: "encryptionkey@0"
I20260620 21:45:22.117202 2028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "7cc758ec7e3d487ba586d3a0fa7e501b"
format_stamp: "Formatted at 2026-06-20 21:45:22 on dist-test-slave-5bvn"
server_key: "064fbf2e3917280f0087009ba5a15e27"
server_key_iv: "44c35f06b2a81b3ecb1ea8469e89740c"
server_key_version: "encryptionkey@0"
I20260620 21:45:22.120882 2028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 21:45:22.123478 2043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:22.124773 2028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:45:22.124929 2028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "7cc758ec7e3d487ba586d3a0fa7e501b"
format_stamp: "Formatted at 2026-06-20 21:45:22 on dist-test-slave-5bvn"
server_key: "064fbf2e3917280f0087009ba5a15e27"
server_key_iv: "44c35f06b2a81b3ecb1ea8469e89740c"
server_key_version: "encryptionkey@0"
I20260620 21:45:22.125051 2028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:22.157671 2028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:22.160799 2028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:22.161012 2028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:22.161624 2028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:22.162508 2028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:22.162581 2028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:22.162667 2028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:22.162721 2028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:22.173036 2028 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:35739
I20260620 21:45:22.173056 2156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:35739 every 8 connection(s)
I20260620 21:45:22.174074 2028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:45:22.183346 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 2028
Jun 20 21:45:22 dist-test-slave-5bvn krb5kdc[1653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991922, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:22.183491 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:45:22.183795 1646 external_mini_cluster.cc:1428] Setting key 2c659504133d02252aad2ab18f8b740d
I20260620 21:45:22.187883 1736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:22.175900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:60351 (local address 127.1.155.190:41197)
0620 21:45:22.176066 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:22.176071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:22.176818 (+ 747us) server_negotiation.cc:407] Connection header received
0620 21:45:22.178222 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:22.178225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:22.178271 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:22.178349 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:22.179454 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:22.179991 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:22.181132 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:22.181341 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:22.181424 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:22.184148 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:22.184151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:22.184153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:22.184187 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:22.186020 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:22.186581 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:22.186587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:22.186589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:22.186671 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:22.186952 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:22.186955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:22.186956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:22.187137 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:22.187214 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:22.187673 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 21:45:22.187768 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 21:45:22.188668 2159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:22.176236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:41197 (local address 127.1.155.131:60351)
0620 21:45:22.176670 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:22.176711 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:22.178015 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:22.178501 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:22.178507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:22.178843 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:22.179310 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:22.179323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:22.180124 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:22.180127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:22.181005 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:22.181014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:22.181507 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:22.181511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:22.181660 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:22.182352 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:22.182369 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:22.183960 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:22.186222 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:22.186227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:22.186238 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:22.186486 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:22.186750 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:22.186752 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:22.186754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:22.186876 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:22.187220 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:22.187225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:22.187440 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 21:45:22.187620 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 21:45:22.187860 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":119,"threads_started":1}
I20260620 21:45:22.189788 2157 heartbeater.cc:347] Connected to a master server at 127.1.155.190:41197
I20260620 21:45:22.190040 2157 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:22.190503 2157 heartbeater.cc:511] Master 127.1.155.190:41197 requested a full tablet report, sending...
I20260620 21:45:22.191488 1698 ts_manager.cc:194] Registered new tserver with Master: 7cc758ec7e3d487ba586d3a0fa7e501b (127.1.155.131:35739)
I20260620 21:45:22.192013 1698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:60351
I20260620 21:45:22.198408 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:45:22 dist-test-slave-5bvn krb5kdc[1653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991922, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:45:22 dist-test-slave-5bvn krb5kdc[1653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991922, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.155.129@KRBTEST.COM
I20260620 21:45:22.224340 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:22.214789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.1.155.129:41843)
0620 21:45:22.215090 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:22.215103 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:22.215126 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:45:22.215196 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:22.215200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:22.215390 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:22.215505 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:22.216133 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:22.216921 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:22.217763 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:22.218258 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:22.218345 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:22.220763 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:22.220773 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:22.220786 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:22.220809 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:22.222549 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:22.223104 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:22.223109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:22.223110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:22.223192 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:22.223457 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:22.223459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:22.223460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:22.223700 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:22.223884 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:22.224055 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 21:45:22.224199 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":86,"threads_started":1}
W20260620 21:45:22.225279 1803 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47338
Jun 20 21:45:22 dist-test-slave-5bvn krb5kdc[1653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:45:22 dist-test-slave-5bvn krb5kdc[1653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.129@KRBTEST.COM
I20260620 21:45:22.251972 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:22.243344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47352 (local address 127.1.155.129:41843)
0620 21:45:22.243496 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:22.243501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:22.243703 (+ 202us) server_negotiation.cc:407] Connection header received
0620 21:45:22.243879 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:22.243887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:22.243929 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:22.244023 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:22.244729 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:22.245249 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:22.246181 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:22.246417 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:22.246529 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:22.248865 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:22.248868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:22.248870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:22.248894 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:22.250349 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:22.250796 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:22.250803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:22.250806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:22.250886 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:22.251162 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:22.251165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:22.251167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:22.251327 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:22.251459 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:22.251688 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 21:45:22.251804 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:45:22.253055 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 1760
I20260620 21:45:22.258692 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 1894
I20260620 21:45:22.264672 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 2028
I20260620 21:45:22.270222 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 1666
2026-06-20T21:45:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2195 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:45:22 dist-test-slave-5bvn krb5kdc[2171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:45:22 dist-test-slave-5bvn krb5kdc[2171](info): set up 2 sockets
Jun 20 21:45:22 dist-test-slave-5bvn krb5kdc[2171](info): commencing operation
krb5kdc: starting...
W20260620 21:45:23.322093 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:45:23 dist-test-slave-5bvn krb5kdc[2171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:45:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:45:23.449805 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:46105
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:46105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:23.562104 2184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:23.562392 2184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:23.562444 2184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:23.566687 2184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:23.566752 2184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:23.566772 2184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:23.566789 2184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:23.566805 2184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:45:23.572630 2184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:46105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:46105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:23.573729 2184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:23.574831 2184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:23.584141 2189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:23.584357 2184 server_base.cc:1034] running on GCE node
W20260620 21:45:23.584434 2190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:23.584141 2192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:23.585309 2184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:23.587357 2184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:23.588593 2184 hybrid_clock.cc:648] HybridClock initialized: now 1781991923588577 us; error 31 us; skew 500 ppm
Jun 20 21:45:23 dist-test-slave-5bvn krb5kdc[2171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:23.591965 2184 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:45:23.593150 2184 webserver.cc:460] Webserver started at http://127.1.155.190:32913/ using document root <none> and password file <none>
I20260620 21:45:23.593755 2184 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:23.593878 2184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:23.594122 2184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:23.596249 2184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "a14472bd107543b9b4f17abfc8c90a2b"
format_stamp: "Formatted at 2026-06-20 21:45:23 on dist-test-slave-5bvn"
server_key: "9415f582f2f84036b72d982a1df3f44f"
server_key_iv: "55dfcd9de956126de23387b8403f18b7"
server_key_version: "encryptionkey@0"
I20260620 21:45:23.596818 2184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "a14472bd107543b9b4f17abfc8c90a2b"
format_stamp: "Formatted at 2026-06-20 21:45:23 on dist-test-slave-5bvn"
server_key: "9415f582f2f84036b72d982a1df3f44f"
server_key_iv: "55dfcd9de956126de23387b8403f18b7"
server_key_version: "encryptionkey@0"
I20260620 21:45:23.600692 2184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 21:45:23.603272 2199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:23.604619 2184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:45:23.604766 2184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "a14472bd107543b9b4f17abfc8c90a2b"
format_stamp: "Formatted at 2026-06-20 21:45:23 on dist-test-slave-5bvn"
server_key: "9415f582f2f84036b72d982a1df3f44f"
server_key_iv: "55dfcd9de956126de23387b8403f18b7"
server_key_version: "encryptionkey@0"
I20260620 21:45:23.604887 2184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:23.636559 2184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:23.639986 2184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:23.640220 2184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:23.647409 2184 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:46105
I20260620 21:45:23.647423 2251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:46105 every 8 connection(s)
I20260620 21:45:23.648365 2184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:23.651131 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:23.656693 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 2184
I20260620 21:45:23.656787 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:23.656844 2252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b: Bootstrap starting.
I20260620 21:45:23.657055 1646 external_mini_cluster.cc:1428] Setting key be3fdfa8d8d26a1c9d07b20037d9de65
I20260620 21:45:23.659400 2252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:23.660140 2252 log.cc:824] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:23.662010 2252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b: No bootstrap required, opened a new log
Jun 20 21:45:23 dist-test-slave-5bvn krb5kdc[2171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:23.665787 2252 raft_consensus.cc:348] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14472bd107543b9b4f17abfc8c90a2b" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46105 } }
I20260620 21:45:23.666087 2252 raft_consensus.cc:374] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:23.666174 2252 raft_consensus.cc:729] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14472bd107543b9b4f17abfc8c90a2b, State: Initialized, Role: FOLLOWER
I20260620 21:45:23.666638 2252 consensus_queue.cc:260] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14472bd107543b9b4f17abfc8c90a2b" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46105 } }
I20260620 21:45:23.666760 2252 raft_consensus.cc:388] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:23.666841 2252 raft_consensus.cc:482] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:23.666936 2252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:23.668012 2252 raft_consensus.cc:504] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14472bd107543b9b4f17abfc8c90a2b" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46105 } }
I20260620 21:45:23.668350 2252 leader_election.cc:302] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a14472bd107543b9b4f17abfc8c90a2b; no voters:
I20260620 21:45:23.668675 2252 leader_election.cc:288] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:23.668797 2257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:23.669025 2257 raft_consensus.cc:686] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [term 1 LEADER]: Becoming Leader. State: Replica: a14472bd107543b9b4f17abfc8c90a2b, State: Running, Role: LEADER
I20260620 21:45:23.669427 2257 consensus_queue.cc:237] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14472bd107543b9b4f17abfc8c90a2b" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46105 } }
I20260620 21:45:23.670001 2252 sys_catalog.cc:564] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:23.671514 2259 sys_catalog.cc:455] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a14472bd107543b9b4f17abfc8c90a2b. Latest consensus state: current_term: 1 leader_uuid: "a14472bd107543b9b4f17abfc8c90a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14472bd107543b9b4f17abfc8c90a2b" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46105 } } }
I20260620 21:45:23.671506 2258 sys_catalog.cc:455] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a14472bd107543b9b4f17abfc8c90a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14472bd107543b9b4f17abfc8c90a2b" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46105 } } }
I20260620 21:45:23.671746 2258 sys_catalog.cc:458] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:23.671937 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:23.658454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39116 (local address 127.1.155.190:46105)
0620 21:45:23.658908 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:23.658923 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:23.658958 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:45:23.659642 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:23.659660 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:23.659988 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:23.660320 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:23.661111 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:23.662194 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:23.663049 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:23.663524 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:23.663693 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:23.666575 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:23.666581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:23.666594 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:23.666628 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:23.668463 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:23.668996 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:23.669005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:23.669014 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:23.669178 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:23.669506 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:23.669510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:23.669512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:23.670079 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:23.670285 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:23.670550 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 21:45:23.670875 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":140,"threads_started":1}
I20260620 21:45:23.671746 2259 sys_catalog.cc:458] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:23.672132 2271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:23.675642 2271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:23.681061 2271 catalog_manager.cc:1347] Generated new cluster ID: 6ba3289cc9c94dfa843442f1535e7ef2
I20260620 21:45:23.681133 2271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:23.692039 2271 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:23.693141 2271 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:23.705221 2271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a14472bd107543b9b4f17abfc8c90a2b: Generated new TSK 0
I20260620 21:45:23.706104 2271 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:45:23.745748 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46105
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:23.883141 2278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:23.883423 2278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:23.883472 2278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:23.887624 2278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:23.887693 2278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:23.887794 2278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:45:23.893777 2278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46105
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:23.894912 2278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:23.895943 2278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:23.905606 2286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:23.905625 2283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:23.905676 2284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:23.905826 2278 server_base.cc:1034] running on GCE node
I20260620 21:45:23.906488 2278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:23.907151 2278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:23.908352 2278 hybrid_clock.cc:648] HybridClock initialized: now 1781991923908338 us; error 28 us; skew 500 ppm
Jun 20 21:45:23 dist-test-slave-5bvn krb5kdc[2171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:23.911170 2278 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:45:23.912056 2278 webserver.cc:460] Webserver started at http://127.1.155.129:37115/ using document root <none> and password file <none>
I20260620 21:45:23.912616 2278 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:23.912705 2278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:23.912935 2278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:23.914996 2278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "e0c5fa7f39e442458795b7fe125812bf"
format_stamp: "Formatted at 2026-06-20 21:45:23 on dist-test-slave-5bvn"
server_key: "9c444ec038cad5051f5f748a3e92deec"
server_key_iv: "372da98a7c654ebc030a749c079ff67a"
server_key_version: "encryptionkey@0"
I20260620 21:45:23.915557 2278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "e0c5fa7f39e442458795b7fe125812bf"
format_stamp: "Formatted at 2026-06-20 21:45:23 on dist-test-slave-5bvn"
server_key: "9c444ec038cad5051f5f748a3e92deec"
server_key_iv: "372da98a7c654ebc030a749c079ff67a"
server_key_version: "encryptionkey@0"
I20260620 21:45:23.919239 2278 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:45:23.921734 2293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:23.923107 2278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:45:23.923269 2278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "e0c5fa7f39e442458795b7fe125812bf"
format_stamp: "Formatted at 2026-06-20 21:45:23 on dist-test-slave-5bvn"
server_key: "9c444ec038cad5051f5f748a3e92deec"
server_key_iv: "372da98a7c654ebc030a749c079ff67a"
server_key_version: "encryptionkey@0"
I20260620 21:45:23.923388 2278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:23.940699 2278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:23.944824 2278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:23.945217 2278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:23.946112 2278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:23.947367 2278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:23.947427 2278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:23.947470 2278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:23.947491 2278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:23.956664 2278 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:44965
I20260620 21:45:23.956688 2406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:44965 every 8 connection(s)
I20260620 21:45:23.957620 2278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:45:23.963433 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 2278
I20260620 21:45:23.963553 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:45:23.963847 1646 external_mini_cluster.cc:1428] Setting key b66e64ea12e0ff2f35755ea014b8f4c6
Jun 20 21:45:23 dist-test-slave-5bvn krb5kdc[2171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:23.973528 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:23.959786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:46815 (local address 127.1.155.190:46105)
0620 21:45:23.959963 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:23.959969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:23.960784 (+ 815us) server_negotiation.cc:407] Connection header received
0620 21:45:23.962209 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:23.962212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:23.962260 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:23.962341 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:23.963729 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:23.964356 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:23.965747 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:23.966045 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:23.966128 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:23.969003 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:23.969010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:23.969017 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:23.969050 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:23.970968 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:23.971724 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:23.971727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:23.971728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:23.971810 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:23.972296 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:23.972299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:23.972300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:23.972510 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:23.972598 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:23.973252 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 21:45:23.973373 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
I20260620 21:45:23.975173 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:23.960135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46105 (local address 127.1.155.129:46815)
0620 21:45:23.960624 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:23.960668 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:23.961980 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:23.962509 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:23.962517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:23.962884 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:23.963548 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:23.963571 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:23.964545 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:23.964548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:23.965606 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:23.965618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:23.966199 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:23.966206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:23.966437 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:23.967087 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:23.967117 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:23.968815 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:23.971102 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:23.971111 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:23.971133 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:23.971615 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:23.971915 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:23.971920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:23.971923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:23.972142 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:23.972624 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:23.972633 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:23.972985 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 21:45:23.973220 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 21:45:23.973923 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
I20260620 21:45:23.976617 2407 heartbeater.cc:347] Connected to a master server at 127.1.155.190:46105
I20260620 21:45:23.976854 2407 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:23.977329 2407 heartbeater.cc:511] Master 127.1.155.190:46105 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
I20260620 21:45:23.979315 2216 ts_manager.cc:194] Registered new tserver with Master: e0c5fa7f39e442458795b7fe125812bf (127.1.155.129:44965)
I20260620 21:45:23.981395 2216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:46815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:45:23.997763 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46105
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:24.115586 2412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:24.115882 2412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:24.115957 2412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:24.120108 2412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:24.120204 2412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:24.120291 2412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:45:24.125628 2412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46105
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:24.126811 2412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:24.127879 2412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:24.137526 2420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:24.137558 2417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:24.137844 2418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:24.138513 2412 server_base.cc:1034] running on GCE node
I20260620 21:45:24.139133 2412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:24.139816 2412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:24.141009 2412 hybrid_clock.cc:648] HybridClock initialized: now 1781991924140978 us; error 54 us; skew 500 ppm
Jun 20 21:45:24 dist-test-slave-5bvn krb5kdc[2171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991924, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:24.144558 2412 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:45:24.145825 2412 webserver.cc:460] Webserver started at http://127.1.155.130:44955/ using document root <none> and password file <none>
I20260620 21:45:24.146509 2412 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:24.146570 2412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:24.146791 2412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:24.148801 2412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "adf46ad0159a4b9c97071d3c37488e4f"
format_stamp: "Formatted at 2026-06-20 21:45:24 on dist-test-slave-5bvn"
server_key: "f8aa3dfdcaa70d4bad77ac325b7d0da4"
server_key_iv: "3d3a8cb964783ccfa5241ff6eb217a0e"
server_key_version: "encryptionkey@0"
I20260620 21:45:24.149370 2412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "adf46ad0159a4b9c97071d3c37488e4f"
format_stamp: "Formatted at 2026-06-20 21:45:24 on dist-test-slave-5bvn"
server_key: "f8aa3dfdcaa70d4bad77ac325b7d0da4"
server_key_iv: "3d3a8cb964783ccfa5241ff6eb217a0e"
server_key_version: "encryptionkey@0"
I20260620 21:45:24.153067 2412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:45:24.155601 2427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:24.156867 2412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:45:24.157015 2412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "adf46ad0159a4b9c97071d3c37488e4f"
format_stamp: "Formatted at 2026-06-20 21:45:24 on dist-test-slave-5bvn"
server_key: "f8aa3dfdcaa70d4bad77ac325b7d0da4"
server_key_iv: "3d3a8cb964783ccfa5241ff6eb217a0e"
server_key_version: "encryptionkey@0"
I20260620 21:45:24.157131 2412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:24.176879 2412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:24.180171 2412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:24.180413 2412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:24.181005 2412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:24.181756 2412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:24.181802 2412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:24.181897 2412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:24.181928 2412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:24.190996 2412 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:39425
I20260620 21:45:24.191006 2540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:39425 every 8 connection(s)
I20260620 21:45:24.191905 2412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:45:24.194576 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 2412
I20260620 21:45:24.194684 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:45:24.194950 1646 external_mini_cluster.cc:1428] Setting key d28017d7e08d2761875d86187157278e
Jun 20 21:45:24 dist-test-slave-5bvn krb5kdc[2171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991924, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
I20260620 21:45:24.206581 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.193784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:38245 (local address 127.1.155.190:46105)
0620 21:45:24.194020 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:24.194024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:24.194878 (+ 854us) server_negotiation.cc:407] Connection header received
0620 21:45:24.196316 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:24.196319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:24.196412 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:24.196513 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:24.197987 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.198514 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.199687 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.199907 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.200023 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.202699 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:24.202702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:24.202704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:24.202727 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:24.204229 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:24.204861 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:24.204865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:24.204866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:24.204947 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:24.205299 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:24.205303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:24.205305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:24.205499 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:24.205617 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:24.206262 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 21:45:24.206390 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260620 21:45:24.207365 2543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.194155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46105 (local address 127.1.155.130:38245)
0620 21:45:24.194690 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:24.194764 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:24.196105 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:24.196686 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:24.196692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:24.197042 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:24.197776 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:24.197793 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.198638 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.198641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:24.199552 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:24.199562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.200058 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.200063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:24.200230 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.200871 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:24.200888 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:24.202515 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:24.204346 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:24.204352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:24.204367 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:24.204736 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:24.205044 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:24.205048 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:24.205050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:24.205191 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:24.205622 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:24.205630 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:24.205972 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 21:45:24.206206 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 21:45:24.206546 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":150,"threads_started":1}
I20260620 21:45:24.208474 2541 heartbeater.cc:347] Connected to a master server at 127.1.155.190:46105
I20260620 21:45:24.208783 2541 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:24.209378 2541 heartbeater.cc:511] Master 127.1.155.190:46105 requested a full tablet report, sending...
I20260620 21:45:24.210433 2216 ts_manager.cc:194] Registered new tserver with Master: adf46ad0159a4b9c97071d3c37488e4f (127.1.155.130:39425)
I20260620 21:45:24.211006 2216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:38245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:45:24.227069 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46105
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:24.339573 2546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:24.339869 2546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:24.339922 2546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:24.344167 2546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:24.344231 2546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:24.344292 2546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:45:24.350240 2546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46105
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:24.351255 2546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:24.352223 2546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:24.361747 2552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:24.361996 2546 server_base.cc:1034] running on GCE node
W20260620 21:45:24.361747 2551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:24.361812 2554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:24.362564 2546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:24.363235 2546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:24.364429 2546 hybrid_clock.cc:648] HybridClock initialized: now 1781991924364404 us; error 40 us; skew 500 ppm
Jun 20 21:45:24 dist-test-slave-5bvn krb5kdc[2171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991924, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:24.367272 2546 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:45:24.368201 2546 webserver.cc:460] Webserver started at http://127.1.155.131:43293/ using document root <none> and password file <none>
I20260620 21:45:24.368772 2546 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:24.368861 2546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:24.369086 2546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:24.371490 2546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "6243bea624f44fc79d95044d84a74c98"
format_stamp: "Formatted at 2026-06-20 21:45:24 on dist-test-slave-5bvn"
server_key: "e192b990e95033ba12b484456aea5f45"
server_key_iv: "65aeb9fc87aab814561564e92d470108"
server_key_version: "encryptionkey@0"
I20260620 21:45:24.372077 2546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "6243bea624f44fc79d95044d84a74c98"
format_stamp: "Formatted at 2026-06-20 21:45:24 on dist-test-slave-5bvn"
server_key: "e192b990e95033ba12b484456aea5f45"
server_key_iv: "65aeb9fc87aab814561564e92d470108"
server_key_version: "encryptionkey@0"
I20260620 21:45:24.375852 2546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 21:45:24.378571 2561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:24.380007 2546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:45:24.380170 2546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "6243bea624f44fc79d95044d84a74c98"
format_stamp: "Formatted at 2026-06-20 21:45:24 on dist-test-slave-5bvn"
server_key: "e192b990e95033ba12b484456aea5f45"
server_key_iv: "65aeb9fc87aab814561564e92d470108"
server_key_version: "encryptionkey@0"
I20260620 21:45:24.380296 2546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:24.413616 2546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:24.416960 2546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:24.417192 2546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:24.417901 2546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:24.418749 2546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:24.418823 2546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:24.418895 2546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:24.418939 2546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:24.428355 2546 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:40071
I20260620 21:45:24.428385 2674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:40071 every 8 connection(s)
I20260620 21:45:24.429260 2546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:45:24.434552 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 2546
I20260620 21:45:24.434696 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:45:24.434954 1646 external_mini_cluster.cc:1428] Setting key cbb893bac37a1990389eae6f40c0756f
Jun 20 21:45:24 dist-test-slave-5bvn krb5kdc[2171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991924, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:24.444116 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.431055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:50215 (local address 127.1.155.190:46105)
0620 21:45:24.431190 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:24.431194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:24.432034 (+ 840us) server_negotiation.cc:407] Connection header received
0620 21:45:24.433875 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:24.433879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:24.433937 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:24.434021 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:24.435177 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.435811 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.437338 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.437576 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.437667 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.440261 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:24.440264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:24.440266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:24.440288 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:24.442146 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:24.442710 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:24.442714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:24.442717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:24.442809 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:24.443143 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:24.443145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:24.443146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:24.443322 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:24.443399 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:24.443879 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 21:45:24.443993 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:45:24.445034 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.431408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46105 (local address 127.1.155.131:50215)
0620 21:45:24.431876 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:24.431918 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:24.433588 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:24.434194 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:24.434201 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:24.434567 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:24.435031 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:24.435044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.436004 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.436008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:24.437216 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:24.437228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.437736 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.437739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:24.437975 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.438600 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:24.438617 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:24.440103 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:24.442270 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:24.442276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:24.442288 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:24.442566 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:24.442910 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:24.442912 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:24.442914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:24.443054 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:24.443410 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:24.443415 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:24.443642 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 21:45:24.443796 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 21:45:24.444145 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":114,"threads_started":1}
I20260620 21:45:24.446256 2675 heartbeater.cc:347] Connected to a master server at 127.1.155.190:46105
I20260620 21:45:24.446539 2675 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:24.447032 2675 heartbeater.cc:511] Master 127.1.155.190:46105 requested a full tablet report, sending...
I20260620 21:45:24.448182 2215 ts_manager.cc:194] Registered new tserver with Master: 6243bea624f44fc79d95044d84a74c98 (127.1.155.131:40071)
I20260620 21:45:24.448920 2215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:50215
I20260620 21:45:24.449050 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:45:24.460386 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.452518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.1.155.190:46105)
0620 21:45:24.452649 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:24.452653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:24.452899 (+ 246us) server_negotiation.cc:407] Connection header received
0620 21:45:24.452975 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:24.452977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:24.453015 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:24.453076 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:24.453905 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.454435 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.455262 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.455477 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.455579 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.457072 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:24.457083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:24.457085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:24.457106 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:24.458729 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:24.459171 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:24.459174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:24.459175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:24.459244 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:24.459530 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:24.459532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:24.459533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:24.459731 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:24.459829 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:24.460033 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 21:45:24.460120 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:45:24.465669 2215 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:45:24.468609 2215 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:24.484136 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.478504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:44965 (local address 127.0.0.1:60254)
0620 21:45:24.479432 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:24.479446 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:24.479529 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:24.480154 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:24.480157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:24.480174 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:24.480279 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:24.480285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.481402 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.481406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:24.482494 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:24.482508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.483543 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.483589 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:24.483693 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.483739 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 21:45:24.483846 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 21:45:24.483982 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":3,"thread_start_us":109,"threads_started":1}
I20260620 21:45:24.484215 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.479227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.1.155.129:44965)
0620 21:45:24.479736 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:24.479741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:24.479763 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:45:24.479833 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:24.479838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:24.480015 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:24.480432 (+ 417us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:24.480546 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.481283 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.482627 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.483413 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.483519 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.483654 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:24.483957 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 21:45:24.484062 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":151,"threads_started":1}
I20260620 21:45:24.484455 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.478506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.1.155.131:40071)
0620 21:45:24.478981 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:24.478995 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:24.479023 (+ 28us) server_negotiation.cc:407] Connection header received
0620 21:45:24.479109 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:24.479115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:24.479318 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:24.479481 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:24.479987 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.481162 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.482739 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.483439 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.483562 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.483698 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:24.484202 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 21:45:24.484315 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":155,"threads_started":1}
I20260620 21:45:24.484658 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.479007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:39425 (local address 127.0.0.1:39788)
0620 21:45:24.479852 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:24.479866 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:24.479951 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:24.480435 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:24.480438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:24.480453 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:24.480532 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:24.480537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.481897 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.481901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:24.483059 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:24.483068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.484224 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.484230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:24.484354 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.484380 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 21:45:24.484453 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 21:45:24.484526 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":31,"thread_start_us":89,"threads_started":1}
I20260620 21:45:24.484720 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.479311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.1.155.130:39425)
0620 21:45:24.479687 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:24.479698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:24.480043 (+ 345us) server_negotiation.cc:407] Connection header received
0620 21:45:24.480116 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:24.480121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:24.480286 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:24.480441 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:24.480647 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.481736 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.483178 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.483952 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.484049 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.484159 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:24.484491 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 21:45:24.484574 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":81,"threads_started":1}
I20260620 21:45:24.484864 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.478320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:40071 (local address 127.0.0.1:57908)
0620 21:45:24.478793 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:24.478847 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:24.478977 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:24.479663 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:24.479669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:24.479694 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:24.479848 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:24.479856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.481298 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.481302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:24.482605 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:24.482615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.483753 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.483757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:24.483897 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.483918 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:45:24.484028 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 21:45:24.484089 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":68,"thread_start_us":87,"threads_started":1}
I20260620 21:45:24.487529 2609 tablet_service.cc:1467] Processing CreateTablet for tablet 1cfb3c951ae044699970b54ff579215a (DEFAULT_TABLE table=test-table [id=f9c8772c6b674236a4396ba06c875ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:24.487556 2341 tablet_service.cc:1467] Processing CreateTablet for tablet 1cfb3c951ae044699970b54ff579215a (DEFAULT_TABLE table=test-table [id=f9c8772c6b674236a4396ba06c875ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:24.487671 2475 tablet_service.cc:1467] Processing CreateTablet for tablet 1cfb3c951ae044699970b54ff579215a (DEFAULT_TABLE table=test-table [id=f9c8772c6b674236a4396ba06c875ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:24.488559 2609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cfb3c951ae044699970b54ff579215a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:24.488560 2475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cfb3c951ae044699970b54ff579215a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:24.488560 2341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cfb3c951ae044699970b54ff579215a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:24.495214 2692 tablet_bootstrap.cc:492] T 1cfb3c951ae044699970b54ff579215a P 6243bea624f44fc79d95044d84a74c98: Bootstrap starting.
I20260620 21:45:24.496148 2694 tablet_bootstrap.cc:492] T 1cfb3c951ae044699970b54ff579215a P adf46ad0159a4b9c97071d3c37488e4f: Bootstrap starting.
I20260620 21:45:24.497691 2692 tablet_bootstrap.cc:654] T 1cfb3c951ae044699970b54ff579215a P 6243bea624f44fc79d95044d84a74c98: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:24.498157 2693 tablet_bootstrap.cc:492] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf: Bootstrap starting.
I20260620 21:45:24.498250 2694 tablet_bootstrap.cc:654] T 1cfb3c951ae044699970b54ff579215a P adf46ad0159a4b9c97071d3c37488e4f: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:24.498607 2692 log.cc:824] T 1cfb3c951ae044699970b54ff579215a P 6243bea624f44fc79d95044d84a74c98: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:24.498936 2694 log.cc:824] T 1cfb3c951ae044699970b54ff579215a P adf46ad0159a4b9c97071d3c37488e4f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:24.500134 2693 tablet_bootstrap.cc:654] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:24.500731 2692 tablet_bootstrap.cc:492] T 1cfb3c951ae044699970b54ff579215a P 6243bea624f44fc79d95044d84a74c98: No bootstrap required, opened a new log
I20260620 21:45:24.500730 2694 tablet_bootstrap.cc:492] T 1cfb3c951ae044699970b54ff579215a P adf46ad0159a4b9c97071d3c37488e4f: No bootstrap required, opened a new log
I20260620 21:45:24.500856 2693 log.cc:824] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:24.500916 2692 ts_tablet_manager.cc:1397] T 1cfb3c951ae044699970b54ff579215a P 6243bea624f44fc79d95044d84a74c98: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:45:24.500916 2694 ts_tablet_manager.cc:1397] T 1cfb3c951ae044699970b54ff579215a P adf46ad0159a4b9c97071d3c37488e4f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 21:45:24.502642 2693 tablet_bootstrap.cc:492] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf: No bootstrap required, opened a new log
I20260620 21:45:24.502832 2693 ts_tablet_manager.cc:1397] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:45:24.504892 2692 raft_consensus.cc:348] T 1cfb3c951ae044699970b54ff579215a P 6243bea624f44fc79d95044d84a74c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6243bea624f44fc79d95044d84a74c98" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40071 } } peers { permanent_uuid: "e0c5fa7f39e442458795b7fe125812bf" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44965 } } peers { permanent_uuid: "adf46ad0159a4b9c97071d3c37488e4f" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39425 } }
I20260620 21:45:24.504909 2694 raft_consensus.cc:348] T 1cfb3c951ae044699970b54ff579215a P adf46ad0159a4b9c97071d3c37488e4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6243bea624f44fc79d95044d84a74c98" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40071 } } peers { permanent_uuid: "e0c5fa7f39e442458795b7fe125812bf" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44965 } } peers { permanent_uuid: "adf46ad0159a4b9c97071d3c37488e4f" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39425 } }
I20260620 21:45:24.505177 2694 raft_consensus.cc:374] T 1cfb3c951ae044699970b54ff579215a P adf46ad0159a4b9c97071d3c37488e4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:24.505177 2692 raft_consensus.cc:374] T 1cfb3c951ae044699970b54ff579215a P 6243bea624f44fc79d95044d84a74c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:24.505232 2694 raft_consensus.cc:729] T 1cfb3c951ae044699970b54ff579215a P adf46ad0159a4b9c97071d3c37488e4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf46ad0159a4b9c97071d3c37488e4f, State: Initialized, Role: FOLLOWER
I20260620 21:45:24.505232 2692 raft_consensus.cc:729] T 1cfb3c951ae044699970b54ff579215a P 6243bea624f44fc79d95044d84a74c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6243bea624f44fc79d95044d84a74c98, State: Initialized, Role: FOLLOWER
I20260620 21:45:24.505689 2692 consensus_queue.cc:260] T 1cfb3c951ae044699970b54ff579215a P 6243bea624f44fc79d95044d84a74c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6243bea624f44fc79d95044d84a74c98" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40071 } } peers { permanent_uuid: "e0c5fa7f39e442458795b7fe125812bf" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44965 } } peers { permanent_uuid: "adf46ad0159a4b9c97071d3c37488e4f" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39425 } }
I20260620 21:45:24.505690 2694 consensus_queue.cc:260] T 1cfb3c951ae044699970b54ff579215a P adf46ad0159a4b9c97071d3c37488e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6243bea624f44fc79d95044d84a74c98" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40071 } } peers { permanent_uuid: "e0c5fa7f39e442458795b7fe125812bf" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44965 } } peers { permanent_uuid: "adf46ad0159a4b9c97071d3c37488e4f" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39425 } }
I20260620 21:45:24.506574 2541 heartbeater.cc:503] Master 127.1.155.190:46105 was elected leader, sending a full tablet report...
I20260620 21:45:24.506704 2694 ts_tablet_manager.cc:1428] T 1cfb3c951ae044699970b54ff579215a P adf46ad0159a4b9c97071d3c37488e4f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:45:24.506704 2692 ts_tablet_manager.cc:1428] T 1cfb3c951ae044699970b54ff579215a P 6243bea624f44fc79d95044d84a74c98: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260620 21:45:24.506928 2693 raft_consensus.cc:348] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6243bea624f44fc79d95044d84a74c98" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40071 } } peers { permanent_uuid: "e0c5fa7f39e442458795b7fe125812bf" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44965 } } peers { permanent_uuid: "adf46ad0159a4b9c97071d3c37488e4f" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39425 } }
I20260620 21:45:24.507148 2693 raft_consensus.cc:374] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:24.507189 2693 raft_consensus.cc:729] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c5fa7f39e442458795b7fe125812bf, State: Initialized, Role: FOLLOWER
I20260620 21:45:24.507612 2693 consensus_queue.cc:260] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6243bea624f44fc79d95044d84a74c98" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40071 } } peers { permanent_uuid: "e0c5fa7f39e442458795b7fe125812bf" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44965 } } peers { permanent_uuid: "adf46ad0159a4b9c97071d3c37488e4f" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39425 } }
I20260620 21:45:24.507767 2675 heartbeater.cc:503] Master 127.1.155.190:46105 was elected leader, sending a full tablet report...
I20260620 21:45:24.508675 2693 ts_tablet_manager.cc:1428] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:45:24.508965 2407 heartbeater.cc:503] Master 127.1.155.190:46105 was elected leader, sending a full tablet report...
I20260620 21:45:24.556334 2700 raft_consensus.cc:482] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:24.556720 2700 raft_consensus.cc:504] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6243bea624f44fc79d95044d84a74c98" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40071 } } peers { permanent_uuid: "e0c5fa7f39e442458795b7fe125812bf" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44965 } } peers { permanent_uuid: "adf46ad0159a4b9c97071d3c37488e4f" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39425 } }
I20260620 21:45:24.557925 2700 leader_election.cc:288] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6243bea624f44fc79d95044d84a74c98 (127.1.155.131:40071), adf46ad0159a4b9c97071d3c37488e4f (127.1.155.130:39425)
I20260620 21:45:24.562592 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.558335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:35565 (local address 127.1.155.131:40071)
0620 21:45:24.558482 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:24.558485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:24.558803 (+ 318us) server_negotiation.cc:407] Connection header received
0620 21:45:24.558887 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:24.558890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:24.558944 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:24.559038 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:24.559345 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.559979 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.561372 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.562080 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.562204 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.562254 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:24.562368 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 21:45:24.562441 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:45:24.562600 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.558222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:40071 (local address 127.1.155.129:35565)
0620 21:45:24.558574 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:24.558604 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:24.558725 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:24.559064 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:24.559067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:24.559086 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:24.559223 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:24.559231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.560083 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.560087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:24.561229 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:24.561239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.562193 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.562196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:24.562266 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.562287 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:45:24.562350 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 21:45:24.562426 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":33,"thread_start_us":129,"threads_started":1}
I20260620 21:45:24.562803 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.558970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:46855 (local address 127.1.155.130:39425)
0620 21:45:24.559217 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:24.559221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:24.559234 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:45:24.559281 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:24.559286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:24.559340 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:24.559416 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:24.559677 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.560230 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.561497 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.562094 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.562204 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.562255 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:24.562584 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 21:45:24.562675 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260620 21:45:24.562948 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.558687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:39425 (local address 127.1.155.129:46855)
0620 21:45:24.559059 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:24.559079 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:24.559148 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:24.559441 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:24.559444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:24.559460 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:24.559571 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:24.559576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.560346 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.560349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:24.561350 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:24.561360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.562327 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.562333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:24.562438 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.562461 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:45:24.562538 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 21:45:24.562600 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":126,"thread_start_us":130,"threads_started":1}
I20260620 21:45:24.563385 2629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfb3c951ae044699970b54ff579215a" candidate_uuid: "e0c5fa7f39e442458795b7fe125812bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6243bea624f44fc79d95044d84a74c98" is_pre_election: true
I20260620 21:45:24.563490 2495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfb3c951ae044699970b54ff579215a" candidate_uuid: "e0c5fa7f39e442458795b7fe125812bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf46ad0159a4b9c97071d3c37488e4f" is_pre_election: true
I20260620 21:45:24.563740 2629 raft_consensus.cc:2454] T 1cfb3c951ae044699970b54ff579215a P 6243bea624f44fc79d95044d84a74c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0c5fa7f39e442458795b7fe125812bf in term 0.
I20260620 21:45:24.563807 2495 raft_consensus.cc:2454] T 1cfb3c951ae044699970b54ff579215a P adf46ad0159a4b9c97071d3c37488e4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0c5fa7f39e442458795b7fe125812bf in term 0.
I20260620 21:45:24.564268 2294 leader_election.cc:302] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6243bea624f44fc79d95044d84a74c98, e0c5fa7f39e442458795b7fe125812bf; no voters:
I20260620 21:45:24.564553 2700 raft_consensus.cc:2781] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:24.564639 2700 raft_consensus.cc:482] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:24.564682 2700 raft_consensus.cc:3037] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:24.565915 2700 raft_consensus.cc:504] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6243bea624f44fc79d95044d84a74c98" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40071 } } peers { permanent_uuid: "e0c5fa7f39e442458795b7fe125812bf" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44965 } } peers { permanent_uuid: "adf46ad0159a4b9c97071d3c37488e4f" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39425 } }
I20260620 21:45:24.566282 2700 leader_election.cc:288] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [CANDIDATE]: Term 1 election: Requested vote from peers 6243bea624f44fc79d95044d84a74c98 (127.1.155.131:40071), adf46ad0159a4b9c97071d3c37488e4f (127.1.155.130:39425)
I20260620 21:45:24.566592 2629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfb3c951ae044699970b54ff579215a" candidate_uuid: "e0c5fa7f39e442458795b7fe125812bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6243bea624f44fc79d95044d84a74c98"
I20260620 21:45:24.566710 2495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfb3c951ae044699970b54ff579215a" candidate_uuid: "e0c5fa7f39e442458795b7fe125812bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf46ad0159a4b9c97071d3c37488e4f"
I20260620 21:45:24.566725 2629 raft_consensus.cc:3037] T 1cfb3c951ae044699970b54ff579215a P 6243bea624f44fc79d95044d84a74c98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:24.566865 2495 raft_consensus.cc:3037] T 1cfb3c951ae044699970b54ff579215a P adf46ad0159a4b9c97071d3c37488e4f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:24.567821 2629 raft_consensus.cc:2454] T 1cfb3c951ae044699970b54ff579215a P 6243bea624f44fc79d95044d84a74c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0c5fa7f39e442458795b7fe125812bf in term 1.
I20260620 21:45:24.567925 2495 raft_consensus.cc:2454] T 1cfb3c951ae044699970b54ff579215a P adf46ad0159a4b9c97071d3c37488e4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0c5fa7f39e442458795b7fe125812bf in term 1.
I20260620 21:45:24.568147 2294 leader_election.cc:302] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6243bea624f44fc79d95044d84a74c98, e0c5fa7f39e442458795b7fe125812bf; no voters:
I20260620 21:45:24.568326 2700 raft_consensus.cc:2781] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:24.568573 2700 raft_consensus.cc:686] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [term 1 LEADER]: Becoming Leader. State: Replica: e0c5fa7f39e442458795b7fe125812bf, State: Running, Role: LEADER
I20260620 21:45:24.569029 2700 consensus_queue.cc:237] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6243bea624f44fc79d95044d84a74c98" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40071 } } peers { permanent_uuid: "e0c5fa7f39e442458795b7fe125812bf" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44965 } } peers { permanent_uuid: "adf46ad0159a4b9c97071d3c37488e4f" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39425 } }
I20260620 21:45:24.572510 2215 catalog_manager.cc:5495] T 1cfb3c951ae044699970b54ff579215a P e0c5fa7f39e442458795b7fe125812bf reported cstate change: term changed from 0 to 1, leader changed from <none> to e0c5fa7f39e442458795b7fe125812bf (127.1.155.129). New cstate: current_term: 1 leader_uuid: "e0c5fa7f39e442458795b7fe125812bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6243bea624f44fc79d95044d84a74c98" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0c5fa7f39e442458795b7fe125812bf" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adf46ad0159a4b9c97071d3c37488e4f" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39425 } health_report { overall_health: UNKNOWN } } }
Jun 20 21:45:24 dist-test-slave-5bvn krb5kdc[2171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.129@KRBTEST.COM
I20260620 21:45:24.587821 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.577059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.1.155.129:44965)
0620 21:45:24.577185 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:24.577195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:24.577411 (+ 216us) server_negotiation.cc:407] Connection header received
0620 21:45:24.577476 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:24.577478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:24.577522 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:24.577594 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:24.578330 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.578827 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.579921 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.580281 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.580392 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.583261 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:24.583285 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:24.583308 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:24.583361 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:24.585970 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:24.586435 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:24.586441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:24.586442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:24.586542 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:24.586820 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:24.586823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:24.586824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:24.587111 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:24.587294 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:24.587548 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 21:45:24.587676 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 21:45:24 dist-test-slave-5bvn krb5kdc[2171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:45:24 dist-test-slave-5bvn krb5kdc[2171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.155.129@KRBTEST.COM
I20260620 21:45:24.616953 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.607655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.1.155.129:44965)
0620 21:45:24.607983 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:24.607987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:24.608198 (+ 211us) server_negotiation.cc:407] Connection header received
0620 21:45:24.608347 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:24.608352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:24.608421 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:24.608502 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:24.609256 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.609881 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.610758 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.611015 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.611104 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.613688 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:24.613692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:24.613694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:24.613718 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:24.615218 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:24.615654 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:24.615659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:24.615661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:24.615744 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:24.616037 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:24.616040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:24.616042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:24.616214 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:24.616387 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:24.616659 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 21:45:24.616777 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 21:45:24 dist-test-slave-5bvn krb5kdc[2171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:45:24 dist-test-slave-5bvn krb5kdc[2171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.129@KRBTEST.COM
I20260620 21:45:24.646392 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:24.635916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60276 (local address 127.1.155.129:44965)
0620 21:45:24.636039 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:24.636042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:24.636115 (+ 73us) server_negotiation.cc:407] Connection header received
0620 21:45:24.636234 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:24.636236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:24.636279 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:24.636357 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:24.637100 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.637668 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.639121 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:24.639480 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:24.639573 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:24.642389 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:24.642392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:24.642394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:24.642417 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:24.643834 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:24.644587 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:24.644592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:24.644594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:24.644741 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:24.645257 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:24.645271 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:24.645274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:24.645490 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:24.645668 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:24.646028 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 21:45:24.646160 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 21:45:24.647997 2321 tablet_service.cc:2908] Rejecting scan request for tablet 1cfb3c951ae044699970b54ff579215a: Uninitialized: clean time has not yet been initialized
I20260620 21:45:24.650460 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 2278
I20260620 21:45:24.658944 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 2412
I20260620 21:45:24.666882 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 2546
I20260620 21:45:24.673453 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 2184
2026-06-20T21:45:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2403 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:45:24 dist-test-slave-5bvn krb5kdc[2717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:45:24 dist-test-slave-5bvn krb5kdc[2717](info): set up 2 sockets
Jun 20 21:45:24 dist-test-slave-5bvn krb5kdc[2717](info): commencing operation
krb5kdc: starting...
W20260620 21:45:25.733583 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:45:25 dist-test-slave-5bvn krb5kdc[2717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:45:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:45:25.857421 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:42155
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:42155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:25.965932 2730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:25.966213 2730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:25.966305 2730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:25.970286 2730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:25.970355 2730 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 21:45:25.970386 2730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:25.970448 2730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:25.970503 2730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:25.970541 2730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:45:25.975862 2730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:42155
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:42155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:25.976943 2730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:25.977981 2730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:25.987370 2736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:25.987370 2738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:25.987409 2735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:25.987663 2730 server_base.cc:1034] running on GCE node
I20260620 21:45:25.988400 2730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:25.989507 2730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:25.990715 2730 hybrid_clock.cc:648] HybridClock initialized: now 1781991925990689 us; error 42 us; skew 500 ppm
Jun 20 21:45:25 dist-test-slave-5bvn krb5kdc[2717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:25.996771 2730 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:45:25.997646 2730 webserver.cc:460] Webserver started at http://127.1.155.190:43033/ using document root <none> and password file <none>
I20260620 21:45:25.998231 2730 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:25.998292 2730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:25.998452 2730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:26.000416 2730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "b24d96881a2f48c585fd18acd951c1f3"
format_stamp: "Formatted at 2026-06-20 21:45:25 on dist-test-slave-5bvn"
server_key: "a3ccbe6cb97b121c8212332f5c5793ac"
server_key_iv: "00078131166064515788dcf84362bf8a"
server_key_version: "encryptionkey@0"
I20260620 21:45:26.000936 2730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "b24d96881a2f48c585fd18acd951c1f3"
format_stamp: "Formatted at 2026-06-20 21:45:25 on dist-test-slave-5bvn"
server_key: "a3ccbe6cb97b121c8212332f5c5793ac"
server_key_iv: "00078131166064515788dcf84362bf8a"
server_key_version: "encryptionkey@0"
I20260620 21:45:26.004714 2730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 21:45:26.007108 2745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:26.008366 2730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:45:26.008472 2730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "b24d96881a2f48c585fd18acd951c1f3"
format_stamp: "Formatted at 2026-06-20 21:45:25 on dist-test-slave-5bvn"
server_key: "a3ccbe6cb97b121c8212332f5c5793ac"
server_key_iv: "00078131166064515788dcf84362bf8a"
server_key_version: "encryptionkey@0"
I20260620 21:45:26.008560 2730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:26.045045 2730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:26.048686 2730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:26.048853 2730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:26.056994 2730 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:42155
I20260620 21:45:26.057011 2807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:42155 every 8 connection(s)
I20260620 21:45:26.058017 2730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:26.060724 2808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:26.063735 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 2730
I20260620 21:45:26.063867 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:26.064129 1646 external_mini_cluster.cc:1428] Setting key 89e6944693513836a8381905767db986
I20260620 21:45:26.066763 2808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3: Bootstrap starting.
I20260620 21:45:26.068920 2808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:26.069646 2808 log.cc:824] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:26.071815 2808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3: No bootstrap required, opened a new log
Jun 20 21:45:26 dist-test-slave-5bvn krb5kdc[2717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:26.075641 2808 raft_consensus.cc:348] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24d96881a2f48c585fd18acd951c1f3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42155 } }
I20260620 21:45:26.075938 2808 raft_consensus.cc:374] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:26.076002 2808 raft_consensus.cc:729] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24d96881a2f48c585fd18acd951c1f3, State: Initialized, Role: FOLLOWER
I20260620 21:45:26.076476 2808 consensus_queue.cc:260] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24d96881a2f48c585fd18acd951c1f3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42155 } }
I20260620 21:45:26.076648 2808 raft_consensus.cc:388] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:26.076735 2808 raft_consensus.cc:482] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:26.076875 2808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:26.078092 2808 raft_consensus.cc:504] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24d96881a2f48c585fd18acd951c1f3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42155 } }
I20260620 21:45:26.078449 2808 leader_election.cc:302] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b24d96881a2f48c585fd18acd951c1f3; no voters:
I20260620 21:45:26.078773 2808 leader_election.cc:288] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:26.078863 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.065589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.1.155.190:42155)
0620 21:45:26.066055 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:26.066070 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:26.066104 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:45:26.066734 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:26.066750 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:26.067136 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:26.067466 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:26.068226 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.069278 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.070243 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.070868 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.071040 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.073911 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:26.073916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:26.073929 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:26.073955 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:26.075673 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.076165 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.076169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.076174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.076333 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.076642 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.076646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.076650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.077170 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:26.077325 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:26.077590 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 21:45:26.077927 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":160,"threads_started":1}
I20260620 21:45:26.078886 2813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:26.079123 2813 raft_consensus.cc:686] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [term 1 LEADER]: Becoming Leader. State: Replica: b24d96881a2f48c585fd18acd951c1f3, State: Running, Role: LEADER
I20260620 21:45:26.079452 2813 consensus_queue.cc:237] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24d96881a2f48c585fd18acd951c1f3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42155 } }
I20260620 21:45:26.080039 2808 sys_catalog.cc:564] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:26.080611 2813 sys_catalog.cc:455] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b24d96881a2f48c585fd18acd951c1f3. Latest consensus state: current_term: 1 leader_uuid: "b24d96881a2f48c585fd18acd951c1f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24d96881a2f48c585fd18acd951c1f3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42155 } } }
I20260620 21:45:26.080611 2814 sys_catalog.cc:455] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b24d96881a2f48c585fd18acd951c1f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24d96881a2f48c585fd18acd951c1f3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42155 } } }
I20260620 21:45:26.080757 2814 sys_catalog.cc:458] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:26.080757 2813 sys_catalog.cc:458] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3 [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:26.081138 2817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:26.084414 2817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:26.090929 2817 catalog_manager.cc:1347] Generated new cluster ID: 8d42e76e69a04f9f97a8fc2e1f34383f
I20260620 21:45:26.091013 2817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:26.130949 2817 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:26.131944 2817 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:26.146121 2817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3: Generated new TSK 0
I20260620 21:45:26.147043 2817 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:45:26.184821 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42155
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:26.294230 2834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:26.294514 2834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:26.294592 2834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:26.298581 2834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:26.298661 2834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:26.298732 2834 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:45:26.298866 2834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:45:26.304127 2834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.155.190:42155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:26.305181 2834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:26.306253 2834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:26.315800 2839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:26.315822 2842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:26.315896 2840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:26.316207 2834 server_base.cc:1034] running on GCE node
I20260620 21:45:26.316618 2834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:26.317361 2834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:26.318557 2834 hybrid_clock.cc:648] HybridClock initialized: now 1781991926318546 us; error 31 us; skew 500 ppm
Jun 20 21:45:26 dist-test-slave-5bvn krb5kdc[2717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991926, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:26.321296 2834 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:45:26.322163 2834 webserver.cc:460] Webserver started at http://127.1.155.129:37027/ using document root <none> and password file <none>
I20260620 21:45:26.322700 2834 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:26.322785 2834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:26.323005 2834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:26.324944 2834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "2125c1d7cdf441da86331d7b1aa2d2d4"
format_stamp: "Formatted at 2026-06-20 21:45:26 on dist-test-slave-5bvn"
server_key: "b4b028ce8d0261f5da1197e2ce3ec796"
server_key_iv: "8ac575107dcc5ebb6a28e1ae297050ab"
server_key_version: "encryptionkey@0"
I20260620 21:45:26.325474 2834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "2125c1d7cdf441da86331d7b1aa2d2d4"
format_stamp: "Formatted at 2026-06-20 21:45:26 on dist-test-slave-5bvn"
server_key: "b4b028ce8d0261f5da1197e2ce3ec796"
server_key_iv: "8ac575107dcc5ebb6a28e1ae297050ab"
server_key_version: "encryptionkey@0"
I20260620 21:45:26.329135 2834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:45:26.331804 2849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:26.333016 2834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 21:45:26.333155 2834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "2125c1d7cdf441da86331d7b1aa2d2d4"
format_stamp: "Formatted at 2026-06-20 21:45:26 on dist-test-slave-5bvn"
server_key: "b4b028ce8d0261f5da1197e2ce3ec796"
server_key_iv: "8ac575107dcc5ebb6a28e1ae297050ab"
server_key_version: "encryptionkey@0"
I20260620 21:45:26.333290 2834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:26.362390 2834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:26.365824 2834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:26.366066 2834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:26.367471 2834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:26.367547 2834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:26.367619 2834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:26.367677 2834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:45:26 dist-test-slave-5bvn krb5kdc[2717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991926, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:26.378851 2834 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:41543
I20260620 21:45:26.379523 2964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:41543 every 8 connection(s)
I20260620 21:45:26.379835 2834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:45:26.381409 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 2834
I20260620 21:45:26.381521 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:45:26.381825 1646 external_mini_cluster.cc:1428] Setting key 9e9a02e4a7284bdff03bbdc8e414edbc
I20260620 21:45:26.381814 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.368465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:59307 (local address 127.1.155.190:42155)
0620 21:45:26.368592 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:26.368595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:26.369610 (+ 1015us) server_negotiation.cc:407] Connection header received
0620 21:45:26.370678 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:26.370682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:26.370739 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:26.370826 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:26.372376 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.373058 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.374580 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.374850 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.374968 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.377786 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:26.377790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:26.377792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:26.377828 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:26.379344 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.379937 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.379942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.379948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.380107 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.380513 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.380516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.380518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.380765 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:26.380863 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:26.381459 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 21:45:26.381625 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:45:26.382747 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.368947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42155 (local address 127.1.155.129:59307)
0620 21:45:26.369437 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:26.369477 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:26.370421 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:26.371094 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:26.371104 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:26.371608 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:26.372223 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.372244 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.373184 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.373188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.374452 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.374461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.375024 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.375028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.375238 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.375889 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:26.375905 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:26.377611 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:26.379459 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:26.379468 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:26.379484 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:26.379810 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:26.380211 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:26.380214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:26.380216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:26.380389 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:26.380853 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:26.380859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:26.381188 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 21:45:26.381413 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 21:45:26.381779 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":140,"threads_started":1}
I20260620 21:45:26.384133 2965 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42155
I20260620 21:45:26.384323 2965 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:26.384795 2965 heartbeater.cc:511] Master 127.1.155.190:42155 requested a full tablet report, sending...
I20260620 21:45:26.386430 2761 ts_manager.cc:194] Registered new tserver with Master: 2125c1d7cdf441da86331d7b1aa2d2d4 (127.1.155.129:41543)
I20260620 21:45:26.387545 2761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:59307
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
I20260620 21:45:26.395367 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.386475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:59317 (local address 127.1.155.190:42155)
0620 21:45:26.386606 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:26.386610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:26.388337 (+ 1727us) server_negotiation.cc:407] Connection header received
0620 21:45:26.388460 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:26.388462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:26.388507 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:26.388595 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:26.389031 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.389523 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.390579 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.390824 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.390937 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.392043 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:26.392046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:26.392049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:26.392079 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:26.393565 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.394092 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.394097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.394098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.394182 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.394463 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.394468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.394470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.394720 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:26.394829 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:26.395080 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 21:45:26.395205 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:45:26.395378 2972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.386418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42155 (local address 127.1.155.129:59317)
0620 21:45:26.388257 (+ 1839us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:26.388269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:26.388377 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:26.388611 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:26.388614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:26.388824 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:26.388918 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.388924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.389658 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.389662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.390307 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.390315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.390939 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.390943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.391043 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.391591 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:26.391602 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:26.391907 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:26.393714 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:26.393720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:26.393725 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:26.393989 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:26.394283 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:26.394286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:26.394288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:26.394372 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:26.394818 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:26.394822 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:26.394921 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 21:45:26.395028 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 21:45:26.395211 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1755,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:45:26.413796 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42155
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:26.522571 2974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:26.522805 2974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:26.522848 2974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:26.526846 2974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:26.526899 2974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:26.526937 2974 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:45:26.526969 2974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:45:26.532140 2974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.155.190:42155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:26.533056 2974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:26.534018 2974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:26.543462 2979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:26.543447 2980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:26.543433 2982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:26.543881 2974 server_base.cc:1034] running on GCE node
I20260620 21:45:26.544307 2974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:26.544924 2974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:26.546110 2974 hybrid_clock.cc:648] HybridClock initialized: now 1781991926546100 us; error 27 us; skew 500 ppm
Jun 20 21:45:26 dist-test-slave-5bvn krb5kdc[2717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991926, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:26.548840 2974 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:45:26.549741 2974 webserver.cc:460] Webserver started at http://127.1.155.130:45765/ using document root <none> and password file <none>
I20260620 21:45:26.550307 2974 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:26.550397 2974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:26.550619 2974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:26.552583 2974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "901625bab3e94ae1a56386bfaa752a28"
format_stamp: "Formatted at 2026-06-20 21:45:26 on dist-test-slave-5bvn"
server_key: "4dba57e9f6b6e0e5504076dd029fa56c"
server_key_iv: "087dacf1fee0d90f07e0311462a9f4c9"
server_key_version: "encryptionkey@0"
I20260620 21:45:26.553135 2974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "901625bab3e94ae1a56386bfaa752a28"
format_stamp: "Formatted at 2026-06-20 21:45:26 on dist-test-slave-5bvn"
server_key: "4dba57e9f6b6e0e5504076dd029fa56c"
server_key_iv: "087dacf1fee0d90f07e0311462a9f4c9"
server_key_version: "encryptionkey@0"
I20260620 21:45:26.556847 2974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:45:26.559363 2989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:26.560531 2974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:45:26.560678 2974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "901625bab3e94ae1a56386bfaa752a28"
format_stamp: "Formatted at 2026-06-20 21:45:26 on dist-test-slave-5bvn"
server_key: "4dba57e9f6b6e0e5504076dd029fa56c"
server_key_iv: "087dacf1fee0d90f07e0311462a9f4c9"
server_key_version: "encryptionkey@0"
I20260620 21:45:26.560793 2974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:26.592619 2974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:26.596076 2974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:26.596288 2974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:26.597748 2974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:26.597829 2974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:26.597914 2974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:26.597965 2974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:45:26 dist-test-slave-5bvn krb5kdc[2717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991926, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:26.608661 2974 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:39043
I20260620 21:45:26.608744 3104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:39043 every 8 connection(s)
I20260620 21:45:26.609687 2974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:45:26.610162 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 2974
I20260620 21:45:26.610271 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:45:26.610528 1646 external_mini_cluster.cc:1428] Setting key 67907dc3dc9ccacf7a6a5cf728b58f46
I20260620 21:45:26.612025 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.598704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:34565 (local address 127.1.155.190:42155)
0620 21:45:26.598850 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:26.598855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:26.599722 (+ 867us) server_negotiation.cc:407] Connection header received
0620 21:45:26.600765 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:26.600768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:26.600822 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:26.600905 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:26.602427 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.603324 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.604722 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.605116 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.605243 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.608023 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:26.608026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:26.608028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:26.608053 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:26.609647 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.610310 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.610315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.610319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.610418 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.610775 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.610778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.610780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.611051 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:26.611156 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:26.611761 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 21:45:26.611910 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:45:26.613024 2998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.599094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42155 (local address 127.1.155.130:34565)
0620 21:45:26.599527 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:26.599598 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:26.600538 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:26.601111 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:26.601121 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:26.601587 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:26.602263 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.602280 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.603459 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.603465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.604591 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.604600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.605284 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.605290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.605499 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.606160 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:26.606175 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:26.607835 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:26.609768 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:26.609774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:26.609788 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:26.610185 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:26.610505 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:26.610508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:26.610509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:26.610666 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:26.611162 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:26.611167 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:26.611462 (+ 295us) client_negotiation.cc:769] Sending connection context
0620 21:45:26.611664 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 21:45:26.611996 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":132,"threads_started":1}
I20260620 21:45:26.614476 3105 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42155
I20260620 21:45:26.614607 3105 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:26.615190 3105 heartbeater.cc:511] Master 127.1.155.190:42155 requested a full tablet report, sending...
I20260620 21:45:26.616611 2762 ts_manager.cc:194] Registered new tserver with Master: 901625bab3e94ae1a56386bfaa752a28 (127.1.155.130:39043)
I20260620 21:45:26.617311 2762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:34565
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
I20260620 21:45:26.625629 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.617585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:34957 (local address 127.1.155.190:42155)
0620 21:45:26.617737 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:26.617744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:26.617761 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:45:26.617807 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:26.617813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:26.617901 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:26.617993 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:26.618677 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.619387 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.620705 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.620908 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.620982 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.622194 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:26.622197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:26.622199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:26.622264 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:26.624012 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.624401 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.624408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.624412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.624509 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.624839 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.624841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.624842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.625036 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:26.625112 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:26.625415 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 21:45:26.625506 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:45:26.625780 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.616640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42155 (local address 127.1.155.130:34957)
0620 21:45:26.616899 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:26.616915 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:26.616999 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:26.618148 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:26.618152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:26.618390 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:26.618525 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.618534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.619526 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.619531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.620535 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.620547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.621011 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.621013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.621088 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.621626 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:26.621637 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:26.622015 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:26.624123 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:26.624125 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:26.624126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:26.624307 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:26.624630 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:26.624636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:26.624640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:26.624743 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:26.625132 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:26.625136 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:26.625245 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 21:45:26.625365 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 21:45:26.625549 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:45:26.641635 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42155
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:26.751621 3114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:26.751874 3114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:26.751919 3114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:26.755911 3114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:26.755973 3114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:26.756014 3114 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:45:26.756117 3114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:45:26.761260 3114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.155.190:42155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:26.762305 3114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:26.763330 3114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:26.773255 3122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:26.773233 3119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:26.773233 3120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:26.773590 3114 server_base.cc:1034] running on GCE node
I20260620 21:45:26.774027 3114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:26.774688 3114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:26.775852 3114 hybrid_clock.cc:648] HybridClock initialized: now 1781991926775823 us; error 45 us; skew 500 ppm
Jun 20 21:45:26 dist-test-slave-5bvn krb5kdc[2717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991926, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:26.778726 3114 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:45:26.779645 3114 webserver.cc:460] Webserver started at http://127.1.155.131:33699/ using document root <none> and password file <none>
I20260620 21:45:26.780244 3114 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:26.780305 3114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:26.780512 3114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:26.782756 3114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "61f3ae311f6e4b4fb89997efb47b9020"
format_stamp: "Formatted at 2026-06-20 21:45:26 on dist-test-slave-5bvn"
server_key: "a467d85a9272f7e8e3dbb53de42c1f2f"
server_key_iv: "5bc3f2503f58be596ef7fd2996cf0622"
server_key_version: "encryptionkey@0"
I20260620 21:45:26.783304 3114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "61f3ae311f6e4b4fb89997efb47b9020"
format_stamp: "Formatted at 2026-06-20 21:45:26 on dist-test-slave-5bvn"
server_key: "a467d85a9272f7e8e3dbb53de42c1f2f"
server_key_iv: "5bc3f2503f58be596ef7fd2996cf0622"
server_key_version: "encryptionkey@0"
I20260620 21:45:26.787374 3114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:45:26.790092 3129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:26.791378 3114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:45:26.791514 3114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "61f3ae311f6e4b4fb89997efb47b9020"
format_stamp: "Formatted at 2026-06-20 21:45:26 on dist-test-slave-5bvn"
server_key: "a467d85a9272f7e8e3dbb53de42c1f2f"
server_key_iv: "5bc3f2503f58be596ef7fd2996cf0622"
server_key_version: "encryptionkey@0"
I20260620 21:45:26.791633 3114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:26.808009 3114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:26.811477 3114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:26.811683 3114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:26.813232 3114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:26.813280 3114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:26.813344 3114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:26.813375 3114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:45:26 dist-test-slave-5bvn krb5kdc[2717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991926, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:26.824239 3114 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:46371
I20260620 21:45:26.824285 3244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:46371 every 8 connection(s)
I20260620 21:45:26.825268 3114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:45:26.827205 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.814062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:50831 (local address 127.1.155.190:42155)
0620 21:45:26.814233 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:26.814238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:26.815241 (+ 1003us) server_negotiation.cc:407] Connection header received
0620 21:45:26.816447 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:26.816451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:26.816513 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:26.816645 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:26.818060 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.818590 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.819751 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.820011 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.820134 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.822990 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:26.822994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:26.822997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:26.823024 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:26.824694 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.825270 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.825273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.825274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.825386 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.825813 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.825816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.825818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.826102 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:26.826205 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:26.826851 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 21:45:26.826980 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 21:45:26.827663 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 3114
I20260620 21:45:26.827790 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:45:26.828038 1646 external_mini_cluster.cc:1428] Setting key 8e4df270b858ddc2c9f19f17ce063505
I20260620 21:45:26.828080 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.814547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42155 (local address 127.1.155.131:50831)
0620 21:45:26.815034 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:26.815095 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:26.816214 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:26.816821 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:26.816831 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:26.817176 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:26.817902 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.817928 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.818730 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.818734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.819623 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.819632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.820186 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.820190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.820384 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.821048 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:26.821074 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:26.822774 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:26.824801 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:26.824807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:26.824819 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:26.825128 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:26.825489 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:26.825493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:26.825495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:26.825674 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:26.826205 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:26.826214 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:26.826559 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 21:45:26.826795 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 21:45:26.827141 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":105,"threads_started":1}
I20260620 21:45:26.829274 3245 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42155
I20260620 21:45:26.829422 3245 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:26.830824 3245 heartbeater.cc:511] Master 127.1.155.190:42155 requested a full tablet report, sending...
I20260620 21:45:26.831866 2761 ts_manager.cc:194] Registered new tserver with Master: 61f3ae311f6e4b4fb89997efb47b9020 (127.1.155.131:46371)
I20260620 21:45:26.832571 2761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:50831
I20260620 21:45:26.839267 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.831888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:36815 (local address 127.1.155.190:42155)
0620 21:45:26.832001 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:26.832005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:26.832261 (+ 256us) server_negotiation.cc:407] Connection header received
0620 21:45:26.832351 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:26.832353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:26.832394 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:26.832472 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:26.832999 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.833541 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.834443 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.834662 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.834768 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.836028 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:26.836032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:26.836034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:26.836078 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:26.837438 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.837958 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.837963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.837965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.838052 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.838361 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.838366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.838368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.838572 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:26.838698 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:26.838974 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 21:45:26.839086 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:45:26.839303 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.831824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42155 (local address 127.1.155.131:36815)
0620 21:45:26.832170 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:26.832185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:26.832272 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:26.832542 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:26.832545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:26.832790 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:26.832886 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.832892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.833676 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.833679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.834326 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.834334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.834770 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.834773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.834865 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.835425 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:26.835442 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:26.835807 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:26.837559 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:26.837566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:26.837568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:26.837781 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:26.838155 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:26.838160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:26.838162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:26.838256 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:26.838684 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:26.838688 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:26.838795 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 21:45:26.838916 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 21:45:26.839081 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
I20260620 21:45:26.842795 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:45:26 dist-test-slave-5bvn krb5kdc[2717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:45:26 dist-test-slave-5bvn krb5kdc[2717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:26.868513 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.859997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.1.155.190:42155)
0620 21:45:26.860130 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:26.860133 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:26.860145 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:45:26.860184 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:26.860187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:26.860226 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:26.860336 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:26.861032 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.861567 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.862518 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.862737 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.862849 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.865403 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:26.865406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:26.865408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:26.865438 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:26.866844 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.867304 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.867310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.867312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.867393 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:26.867684 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:26.867689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:26.867691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:26.867864 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:26.867997 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:26.868236 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 21:45:26.868347 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:45:26.871367 2761 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:45:26.874365 2761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:26.888917 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.883505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46644 (local address 127.1.155.130:39043)
0620 21:45:26.883915 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:26.883928 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:26.884329 (+ 401us) server_negotiation.cc:407] Connection header received
0620 21:45:26.884401 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:26.884406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:26.884567 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:26.884725 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:26.885120 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.885962 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.887479 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.888241 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.888356 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.888513 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:26.888680 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 21:45:26.888770 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":104,"threads_started":1}
I20260620 21:45:26.888904 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.883339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:39043 (local address 127.0.0.1:46644)
0620 21:45:26.884155 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:26.884168 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:26.884244 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:26.884870 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:26.884873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:26.884896 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:26.885008 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.885013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.886095 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.886098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.887357 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.887366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.888379 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.888386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.888484 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.888519 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 21:45:26.888641 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 21:45:26.888758 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":188,"thread_start_us":84,"threads_started":1}
I20260620 21:45:26.889132 3266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.882720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:46371 (local address 127.0.0.1:60444)
0620 21:45:26.883727 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:26.883779 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:26.883914 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:26.884533 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:26.884539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:26.884561 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:26.884687 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.884693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.886240 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.886242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.887637 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.887648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.888592 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.888596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.888679 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.888701 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:45:26.888773 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 21:45:26.888842 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":37,"thread_start_us":104,"threads_started":1}
I20260620 21:45:26.889453 3265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.882865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.1.155.131:46371)
0620 21:45:26.883477 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:26.883488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:26.884065 (+ 577us) server_negotiation.cc:407] Connection header received
0620 21:45:26.884142 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:26.884147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:26.884384 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:26.884523 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:26.884824 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.886123 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.887781 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.888289 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.888414 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.888563 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:26.888972 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 21:45:26.889071 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":78,"threads_started":1}
I20260620 21:45:26.889923 3264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.882545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:41543 (local address 127.0.0.1:47856)
0620 21:45:26.883726 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:26.883779 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:26.883916 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:26.884623 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:26.884626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:26.884638 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:26.884750 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.884760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.886230 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.886233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.887643 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.887648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.888797 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.888800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.888882 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.888903 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:45:26.888971 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 21:45:26.889220 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":98,"thread_start_us":71,"threads_started":1}
I20260620 21:45:26.890182 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.882700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.1.155.129:41543)
0620 21:45:26.883272 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:26.883289 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:26.883865 (+ 576us) server_negotiation.cc:407] Connection header received
0620 21:45:26.884071 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:26.884083 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:26.884303 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:26.884475 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:26.884869 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.886093 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.887762 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.888506 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.888602 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.888740 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:26.889930 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0620 21:45:26.890026 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":92,"threads_started":1}
I20260620 21:45:26.891980 3039 tablet_service.cc:1467] Processing CreateTablet for tablet a70f88944e9f42c792525b51c82ad0e7 (DEFAULT_TABLE table=test-table [id=47ac6a9abc724c33a930bb3f5cee6c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:26.892154 3179 tablet_service.cc:1467] Processing CreateTablet for tablet a70f88944e9f42c792525b51c82ad0e7 (DEFAULT_TABLE table=test-table [id=47ac6a9abc724c33a930bb3f5cee6c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:26.892971 3039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70f88944e9f42c792525b51c82ad0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:26.892706 2899 tablet_service.cc:1467] Processing CreateTablet for tablet a70f88944e9f42c792525b51c82ad0e7 (DEFAULT_TABLE table=test-table [id=47ac6a9abc724c33a930bb3f5cee6c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:26.893123 3179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70f88944e9f42c792525b51c82ad0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:26.893579 2899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70f88944e9f42c792525b51c82ad0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:26.899760 3268 tablet_bootstrap.cc:492] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28: Bootstrap starting.
I20260620 21:45:26.901655 3269 tablet_bootstrap.cc:492] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4: Bootstrap starting.
I20260620 21:45:26.902057 3270 tablet_bootstrap.cc:492] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020: Bootstrap starting.
I20260620 21:45:26.903156 3268 tablet_bootstrap.cc:654] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:26.904021 3268 log.cc:824] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:26.904105 3270 tablet_bootstrap.cc:654] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:26.904845 3270 log.cc:824] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:26.906138 3268 tablet_bootstrap.cc:492] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28: No bootstrap required, opened a new log
I20260620 21:45:26.906330 3268 ts_tablet_manager.cc:1397] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 21:45:26.907244 3269 tablet_bootstrap.cc:654] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:26.908423 3269 log.cc:824] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:26.910169 3268 raft_consensus.cc:348] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } }
I20260620 21:45:26.910352 3270 tablet_bootstrap.cc:492] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020: No bootstrap required, opened a new log
I20260620 21:45:26.910346 3269 tablet_bootstrap.cc:492] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4: No bootstrap required, opened a new log
I20260620 21:45:26.910406 3268 raft_consensus.cc:374] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:26.910476 3268 raft_consensus.cc:729] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901625bab3e94ae1a56386bfaa752a28, State: Initialized, Role: FOLLOWER
I20260620 21:45:26.910521 3270 ts_tablet_manager.cc:1397] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260620 21:45:26.910521 3269 ts_tablet_manager.cc:1397] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20260620 21:45:26.910913 3268 consensus_queue.cc:260] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } }
I20260620 21:45:26.911573 3105 heartbeater.cc:503] Master 127.1.155.190:42155 was elected leader, sending a full tablet report...
I20260620 21:45:26.911885 3268 ts_tablet_manager.cc:1428] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 21:45:26.914465 3269 raft_consensus.cc:348] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } }
I20260620 21:45:26.914726 3269 raft_consensus.cc:374] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:26.914824 3269 raft_consensus.cc:729] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2125c1d7cdf441da86331d7b1aa2d2d4, State: Initialized, Role: FOLLOWER
I20260620 21:45:26.915110 3270 raft_consensus.cc:348] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } }
I20260620 21:45:26.915241 3269 consensus_queue.cc:260] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } }
I20260620 21:45:26.915343 3270 raft_consensus.cc:374] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:26.915388 3270 raft_consensus.cc:729] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f3ae311f6e4b4fb89997efb47b9020, State: Initialized, Role: FOLLOWER
I20260620 21:45:26.915782 3270 consensus_queue.cc:260] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } }
I20260620 21:45:26.916153 3269 ts_tablet_manager.cc:1428] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:45:26.916200 2965 heartbeater.cc:503] Master 127.1.155.190:42155 was elected leader, sending a full tablet report...
I20260620 21:45:26.916589 3245 heartbeater.cc:503] Master 127.1.155.190:42155 was elected leader, sending a full tablet report...
I20260620 21:45:26.916688 3270 ts_tablet_manager.cc:1428] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260620 21:45:26.933148 3274 raft_consensus.cc:482] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:26.933343 3274 raft_consensus.cc:504] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } }
I20260620 21:45:26.934486 3274 leader_election.cc:288] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2125c1d7cdf441da86331d7b1aa2d2d4 (127.1.155.129:41543), 61f3ae311f6e4b4fb89997efb47b9020 (127.1.155.131:46371)
I20260620 21:45:26.939126 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.934760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:35391 (local address 127.1.155.129:41543)
0620 21:45:26.934879 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:26.934884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:26.934971 (+ 87us) server_negotiation.cc:407] Connection header received
0620 21:45:26.935096 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:26.935099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:26.935165 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:26.935250 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:26.935531 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.936370 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.938037 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.938582 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.938701 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.938751 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:26.938908 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 21:45:26.938994 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:45:26.939136 2998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.934697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:41543 (local address 127.1.155.130:35391)
0620 21:45:26.934886 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:26.934901 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:26.934994 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:26.935272 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:26.935274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:26.935286 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:26.935424 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.935431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.936479 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.936482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.937916 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.937927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.938699 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.938701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.938793 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.938812 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 21:45:26.938872 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 21:45:26.938929 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":2}
I20260620 21:45:26.939507 3265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.934810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:51469 (local address 127.1.155.131:46371)
0620 21:45:26.934944 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:26.934950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:26.935164 (+ 214us) server_negotiation.cc:407] Connection header received
0620 21:45:26.935219 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:26.935224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:26.935280 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:26.935349 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:26.935589 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.936362 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.938300 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.938908 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.939015 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.939057 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:26.939290 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 21:45:26.939379 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:45:26.939850 3277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.934728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:46371 (local address 127.1.155.130:51469)
0620 21:45:26.935067 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:26.935080 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:26.935161 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:26.935367 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:26.935369 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:26.935385 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:26.935489 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.935497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.936479 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.936482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.938195 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:26.938202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.939032 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.939035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:26.939129 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.939161 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 21:45:26.939646 (+ 485us) client_negotiation.cc:240] Negotiation successful
0620 21:45:26.939713 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":33,"thread_start_us":105,"threads_started":1}
I20260620 21:45:26.939775 2919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70f88944e9f42c792525b51c82ad0e7" candidate_uuid: "901625bab3e94ae1a56386bfaa752a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" is_pre_election: true
I20260620 21:45:26.940095 2919 raft_consensus.cc:2454] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 901625bab3e94ae1a56386bfaa752a28 in term 0.
I20260620 21:45:26.940526 3199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70f88944e9f42c792525b51c82ad0e7" candidate_uuid: "901625bab3e94ae1a56386bfaa752a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f3ae311f6e4b4fb89997efb47b9020" is_pre_election: true
I20260620 21:45:26.940601 2992 leader_election.cc:302] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2125c1d7cdf441da86331d7b1aa2d2d4, 901625bab3e94ae1a56386bfaa752a28; no voters:
I20260620 21:45:26.940809 3199 raft_consensus.cc:2454] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 901625bab3e94ae1a56386bfaa752a28 in term 0.
I20260620 21:45:26.940897 3274 raft_consensus.cc:2781] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:26.940984 3274 raft_consensus.cc:482] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:26.941036 3274 raft_consensus.cc:3037] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:26.942081 3274 raft_consensus.cc:504] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } }
I20260620 21:45:26.942358 3274 leader_election.cc:288] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [CANDIDATE]: Term 1 election: Requested vote from peers 2125c1d7cdf441da86331d7b1aa2d2d4 (127.1.155.129:41543), 61f3ae311f6e4b4fb89997efb47b9020 (127.1.155.131:46371)
I20260620 21:45:26.942714 3199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70f88944e9f42c792525b51c82ad0e7" candidate_uuid: "901625bab3e94ae1a56386bfaa752a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f3ae311f6e4b4fb89997efb47b9020"
I20260620 21:45:26.942842 3199 raft_consensus.cc:3037] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:26.944147 3199 raft_consensus.cc:2454] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 901625bab3e94ae1a56386bfaa752a28 in term 1.
I20260620 21:45:26.944430 2919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70f88944e9f42c792525b51c82ad0e7" candidate_uuid: "901625bab3e94ae1a56386bfaa752a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4"
I20260620 21:45:26.944504 2993 leader_election.cc:302] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f3ae311f6e4b4fb89997efb47b9020, 901625bab3e94ae1a56386bfaa752a28; no voters:
I20260620 21:45:26.944548 2919 raft_consensus.cc:3037] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:26.944741 3274 raft_consensus.cc:2781] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:26.945052 3274 raft_consensus.cc:686] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [term 1 LEADER]: Becoming Leader. State: Replica: 901625bab3e94ae1a56386bfaa752a28, State: Running, Role: LEADER
I20260620 21:45:26.945562 3274 consensus_queue.cc:237] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } }
I20260620 21:45:26.945884 2919 raft_consensus.cc:2454] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 901625bab3e94ae1a56386bfaa752a28 in term 1.
I20260620 21:45:26.948786 2761 catalog_manager.cc:5495] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 901625bab3e94ae1a56386bfaa752a28 (127.1.155.130). New cstate: current_term: 1 leader_uuid: "901625bab3e94ae1a56386bfaa752a28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } health_report { overall_health: HEALTHY } } }
I20260620 21:45:26.979388 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:26.975304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46660 (local address 127.1.155.130:39043)
0620 21:45:26.975434 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:26.975437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:26.975450 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:45:26.975602 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:26.975604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:26.975656 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:26.975743 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:26.976154 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.976666 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.977571 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:26.977974 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:26.978060 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:26.978478 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:26.978910 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:26.979012 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:26.979216 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 21:45:26.979274 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:45:26.985790 2919 raft_consensus.cc:1261] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 1 FOLLOWER]: Refusing update from remote peer 901625bab3e94ae1a56386bfaa752a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:26.986038 3199 raft_consensus.cc:1261] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020 [term 1 FOLLOWER]: Refusing update from remote peer 901625bab3e94ae1a56386bfaa752a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:26.986686 3278 consensus_queue.cc:1035] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:26.986843 3274 consensus_queue.cc:1035] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:26.996279 3282 mvcc.cc:205] Tried to move back new op lower bound from 7299038932926402560 to 7299038932772917248. Current Snapshot: MvccSnapshot[applied={T|T < 7299038932926402560}]
I20260620 21:45:26.998876 3281 mvcc.cc:205] Tried to move back new op lower bound from 7299038932926402560 to 7299038932772917248. Current Snapshot: MvccSnapshot[applied={T|T < 7299038932926402560 or (T in {7299038932926402560})}]
I20260620 21:45:27.006642 2761 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:27.006803 2761 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:27.009378 2761 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3: Sending DeleteTablet for 3 replicas of tablet a70f88944e9f42c792525b51c82ad0e7
I20260620 21:45:27.010228 2899 tablet_service.cc:1514] Processing DeleteTablet for tablet a70f88944e9f42c792525b51c82ad0e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:27 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:47856
I20260620 21:45:27.010228 3179 tablet_service.cc:1514] Processing DeleteTablet for tablet a70f88944e9f42c792525b51c82ad0e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:27 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:60444
I20260620 21:45:27.010378 3039 tablet_service.cc:1514] Processing DeleteTablet for tablet a70f88944e9f42c792525b51c82ad0e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:27 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:46644
I20260620 21:45:27.010746 3291 tablet_replica.cc:330] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020: stopping tablet replica
I20260620 21:45:27.010890 3292 tablet_replica.cc:330] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4: stopping tablet replica
I20260620 21:45:27.011256 3293 tablet_replica.cc:330] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28: stopping tablet replica
I20260620 21:45:27.011240 3292 raft_consensus.cc:2229] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:27.011595 3292 raft_consensus.cc:2258] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:27.011624 3293 raft_consensus.cc:2229] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:45:27.011873 2761 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 21:45:27.012257 3293 raft_consensus.cc:2258] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 21:45:27.012336 2761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:27.013180 3292 ts_tablet_manager.cc:1905] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:27.013197 3291 raft_consensus.cc:2229] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:27.013495 3291 raft_consensus.cc:2258] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:27.015787 3291 ts_tablet_manager.cc:1905] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:27.016038 3293 ts_tablet_manager.cc:1905] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:27.016582 3292 ts_tablet_manager.cc:1918] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:45:27.016691 3292 log.cc:1196] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/wal/wals/a70f88944e9f42c792525b51c82ad0e7
I20260620 21:45:27.017030 3292 ts_tablet_manager.cc:1939] T a70f88944e9f42c792525b51c82ad0e7 P 2125c1d7cdf441da86331d7b1aa2d2d4: Deleting consensus metadata
I20260620 21:45:27.018388 3039 tablet_service.cc:1467] Processing CreateTablet for tablet 6bea584d39a743b3b3fabd24169faf55 (DEFAULT_TABLE table=test-table [id=38a162237f994bc2aa94c75f699b83f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:27.018472 2899 tablet_service.cc:1467] Processing CreateTablet for tablet 6bea584d39a743b3b3fabd24169faf55 (DEFAULT_TABLE table=test-table [id=38a162237f994bc2aa94c75f699b83f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:27.018553 3291 ts_tablet_manager.cc:1918] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:45:27.018644 3291 log.cc:1196] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/a70f88944e9f42c792525b51c82ad0e7
I20260620 21:45:27.018676 3039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bea584d39a743b3b3fabd24169faf55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:27.018676 2899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bea584d39a743b3b3fabd24169faf55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:27.018960 3291 ts_tablet_manager.cc:1939] T a70f88944e9f42c792525b51c82ad0e7 P 61f3ae311f6e4b4fb89997efb47b9020: Deleting consensus metadata
I20260620 21:45:27.019609 3293 ts_tablet_manager.cc:1918] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:45:27.019830 3293 log.cc:1196] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/a70f88944e9f42c792525b51c82ad0e7
I20260620 21:45:27.020185 3293 ts_tablet_manager.cc:1939] T a70f88944e9f42c792525b51c82ad0e7 P 901625bab3e94ae1a56386bfaa752a28: Deleting consensus metadata
I20260620 21:45:27.020416 3179 tablet_service.cc:1467] Processing CreateTablet for tablet 6bea584d39a743b3b3fabd24169faf55 (DEFAULT_TABLE table=test-table [id=38a162237f994bc2aa94c75f699b83f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:27.020660 3179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bea584d39a743b3b3fabd24169faf55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:27.020790 3269 tablet_bootstrap.cc:492] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4: Bootstrap starting.
I20260620 21:45:27.020792 2748 catalog_manager.cc:4841] TS 2125c1d7cdf441da86331d7b1aa2d2d4 (127.1.155.129:41543): tablet a70f88944e9f42c792525b51c82ad0e7 (table test-table [id=47ac6a9abc724c33a930bb3f5cee6c78]) successfully deleted
I20260620 21:45:27.021351 3268 tablet_bootstrap.cc:492] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28: Bootstrap starting.
I20260620 21:45:27.021934 3269 tablet_bootstrap.cc:654] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:27.022363 3268 tablet_bootstrap.cc:654] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:27.022820 3270 tablet_bootstrap.cc:492] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020: Bootstrap starting.
I20260620 21:45:27.022957 2749 catalog_manager.cc:4841] TS 61f3ae311f6e4b4fb89997efb47b9020 (127.1.155.131:46371): tablet a70f88944e9f42c792525b51c82ad0e7 (table test-table [id=47ac6a9abc724c33a930bb3f5cee6c78]) successfully deleted
I20260620 21:45:27.023334 3269 tablet_bootstrap.cc:492] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4: No bootstrap required, opened a new log
I20260620 21:45:27.023407 3269 ts_tablet_manager.cc:1397] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 21:45:27.023748 3270 tablet_bootstrap.cc:654] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:27.023880 2746 catalog_manager.cc:4841] TS 901625bab3e94ae1a56386bfaa752a28 (127.1.155.130:39043): tablet a70f88944e9f42c792525b51c82ad0e7 (table test-table [id=47ac6a9abc724c33a930bb3f5cee6c78]) successfully deleted
I20260620 21:45:27.024293 3269 raft_consensus.cc:348] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.024436 3269 raft_consensus.cc:374] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:27.024487 3269 raft_consensus.cc:729] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2125c1d7cdf441da86331d7b1aa2d2d4, State: Initialized, Role: FOLLOWER
I20260620 21:45:27.024618 3269 consensus_queue.cc:260] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.024726 3268 tablet_bootstrap.cc:492] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28: No bootstrap required, opened a new log
I20260620 21:45:27.024801 3268 ts_tablet_manager.cc:1397] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:45:27.024886 3269 ts_tablet_manager.cc:1428] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260620 21:45:27.025274 3268 raft_consensus.cc:348] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.025379 3268 raft_consensus.cc:374] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:27.025413 3268 raft_consensus.cc:729] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901625bab3e94ae1a56386bfaa752a28, State: Initialized, Role: FOLLOWER
I20260620 21:45:27.025524 3268 consensus_queue.cc:260] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.025772 3268 ts_tablet_manager.cc:1428] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:27.026952 3270 tablet_bootstrap.cc:492] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020: No bootstrap required, opened a new log
I20260620 21:45:27.027045 3270 ts_tablet_manager.cc:1397] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260620 21:45:27.027472 3270 raft_consensus.cc:348] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.027583 3270 raft_consensus.cc:374] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:27.027616 3270 raft_consensus.cc:729] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f3ae311f6e4b4fb89997efb47b9020, State: Initialized, Role: FOLLOWER
I20260620 21:45:27.027732 3270 consensus_queue.cc:260] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.027988 3270 ts_tablet_manager.cc:1428] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260620 21:45:27.076426 3246 tablet.cc:2369] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:45:27.111617 3106 tablet.cc:2369] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:45:27.131234 2966 tablet.cc:2369] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:45:27.165421 3276 raft_consensus.cc:482] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:27.165566 3276 raft_consensus.cc:504] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.166790 3276 leader_election.cc:288] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2125c1d7cdf441da86331d7b1aa2d2d4 (127.1.155.129:41543), 901625bab3e94ae1a56386bfaa752a28 (127.1.155.130:39043)
I20260620 21:45:27.172292 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:27.167257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:38027 (local address 127.1.155.130:39043)
0620 21:45:27.167376 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:27.167380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:27.167553 (+ 173us) server_negotiation.cc:407] Connection header received
0620 21:45:27.167643 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:27.167646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:27.167689 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:27.167768 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:27.168043 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.168820 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.170816 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.171471 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.171587 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:27.171642 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:27.172068 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 21:45:27.172164 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:45:27.172292 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:27.167165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:43029 (local address 127.1.155.129:41543)
0620 21:45:27.167428 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:27.167432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:27.167444 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:45:27.167488 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:27.167491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:27.167559 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:27.167642 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:27.168021 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.168806 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.170816 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.171419 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.171587 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:27.171642 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:27.172039 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 21:45:27.172165 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260620 21:45:27.172300 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:27.167007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:41543 (local address 127.1.155.131:43029)
0620 21:45:27.167141 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:27.167157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:27.167256 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:27.167688 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:27.167692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:27.167714 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:27.167867 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:27.167876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.168933 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.168937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:27.170651 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:27.170666 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.171607 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.171614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:27.171833 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:27.171878 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 21:45:27.171972 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 21:45:27.172110 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":214}
I20260620 21:45:27.172315 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:27.167130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:39043 (local address 127.1.155.131:38027)
0620 21:45:27.167444 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:27.167457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:27.167559 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:27.167785 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:27.167788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:27.167800 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:27.167933 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:27.167942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.168933 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.168937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:27.170651 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:27.170666 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.171619 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.171623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:27.171823 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:27.171883 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 21:45:27.171972 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 21:45:27.172110 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":275,"thread_start_us":100,"threads_started":1}
I20260620 21:45:27.172871 2919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bea584d39a743b3b3fabd24169faf55" candidate_uuid: "61f3ae311f6e4b4fb89997efb47b9020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" is_pre_election: true
I20260620 21:45:27.173074 2919 raft_consensus.cc:2454] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61f3ae311f6e4b4fb89997efb47b9020 in term 0.
I20260620 21:45:27.173228 3059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bea584d39a743b3b3fabd24169faf55" candidate_uuid: "61f3ae311f6e4b4fb89997efb47b9020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901625bab3e94ae1a56386bfaa752a28" is_pre_election: true
I20260620 21:45:27.173489 3059 raft_consensus.cc:2454] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61f3ae311f6e4b4fb89997efb47b9020 in term 0.
I20260620 21:45:27.173645 3132 leader_election.cc:302] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2125c1d7cdf441da86331d7b1aa2d2d4, 61f3ae311f6e4b4fb89997efb47b9020; no voters:
I20260620 21:45:27.173986 3276 raft_consensus.cc:2781] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:27.174081 3276 raft_consensus.cc:482] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:27.174130 3276 raft_consensus.cc:3037] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:27.175115 3276 raft_consensus.cc:504] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.175499 3276 leader_election.cc:288] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [CANDIDATE]: Term 1 election: Requested vote from peers 2125c1d7cdf441da86331d7b1aa2d2d4 (127.1.155.129:41543), 901625bab3e94ae1a56386bfaa752a28 (127.1.155.130:39043)
I20260620 21:45:27.175859 2919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bea584d39a743b3b3fabd24169faf55" candidate_uuid: "61f3ae311f6e4b4fb89997efb47b9020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4"
I20260620 21:45:27.175905 3059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bea584d39a743b3b3fabd24169faf55" candidate_uuid: "61f3ae311f6e4b4fb89997efb47b9020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901625bab3e94ae1a56386bfaa752a28"
I20260620 21:45:27.176004 2919 raft_consensus.cc:3037] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:27.176033 3059 raft_consensus.cc:3037] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:27.176891 2919 raft_consensus.cc:2454] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61f3ae311f6e4b4fb89997efb47b9020 in term 1.
I20260620 21:45:27.177073 3059 raft_consensus.cc:2454] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61f3ae311f6e4b4fb89997efb47b9020 in term 1.
I20260620 21:45:27.177168 3132 leader_election.cc:302] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2125c1d7cdf441da86331d7b1aa2d2d4, 61f3ae311f6e4b4fb89997efb47b9020; no voters:
I20260620 21:45:27.177356 3276 raft_consensus.cc:2781] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:27.177603 3276 raft_consensus.cc:686] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [term 1 LEADER]: Becoming Leader. State: Replica: 61f3ae311f6e4b4fb89997efb47b9020, State: Running, Role: LEADER
I20260620 21:45:27.177933 3276 consensus_queue.cc:237] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.180346 2761 catalog_manager.cc:5495] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61f3ae311f6e4b4fb89997efb47b9020 (127.1.155.131). New cstate: current_term: 1 leader_uuid: "61f3ae311f6e4b4fb89997efb47b9020" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } health_report { overall_health: HEALTHY } } }
Jun 20 21:45:27 dist-test-slave-5bvn krb5kdc[2717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:27.240351 3305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:27.231644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42155 (local address 127.0.0.1:58730)
0620 21:45:27.231943 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:27.231959 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:27.232090 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:27.232356 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:27.232359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:27.232768 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:27.232857 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:27.232863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.233774 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.233780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:27.234404 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:27.234414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.234881 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.234885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:27.234969 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:27.235517 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:27.235530 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:27.237056 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:27.238971 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:27.238979 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:27.238984 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:27.239246 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:27.239482 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:27.239484 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:27.239485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:27.239563 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:27.239887 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:27.239892 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:27.239989 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 21:45:27.240103 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 21:45:27.240217 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":102,"threads_started":1}
I20260620 21:45:27.240437 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:27.231688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.1.155.190:42155)
0620 21:45:27.231843 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:27.231848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:27.232042 (+ 194us) server_negotiation.cc:407] Connection header received
0620 21:45:27.232197 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:27.232200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:27.232249 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:27.232338 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:27.232990 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.233629 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.234539 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.234753 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.234868 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:27.237312 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:27.237318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:27.237322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:27.237356 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:27.238863 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:27.239334 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:27.239337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:27.239338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:27.239406 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:27.239636 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:27.239639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:27.239640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:27.239805 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:27.239876 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:27.240190 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 21:45:27.240290 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:45:27.244320 2761 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:58730:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 21:45:27.245043 2761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:27.249953 2899 tablet_service.cc:1467] Processing CreateTablet for tablet cb2e716c5c304fb5a70a5174a83b94a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1da767ce423b4101b2feb04c554ff435]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:45:27.249977 3039 tablet_service.cc:1467] Processing CreateTablet for tablet cb2e716c5c304fb5a70a5174a83b94a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1da767ce423b4101b2feb04c554ff435]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:45:27.250131 3179 tablet_service.cc:1467] Processing CreateTablet for tablet cb2e716c5c304fb5a70a5174a83b94a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1da767ce423b4101b2feb04c554ff435]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:45:27.250360 3039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2e716c5c304fb5a70a5174a83b94a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:27.250361 2899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2e716c5c304fb5a70a5174a83b94a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:27.250573 3179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2e716c5c304fb5a70a5174a83b94a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:27.252835 3270 tablet_bootstrap.cc:492] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020: Bootstrap starting.
I20260620 21:45:27.252957 3268 tablet_bootstrap.cc:492] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28: Bootstrap starting.
I20260620 21:45:27.253002 3269 tablet_bootstrap.cc:492] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4: Bootstrap starting.
I20260620 21:45:27.253819 3268 tablet_bootstrap.cc:654] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:27.253865 3270 tablet_bootstrap.cc:654] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:27.253906 3269 tablet_bootstrap.cc:654] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:27.255159 3268 tablet_bootstrap.cc:492] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28: No bootstrap required, opened a new log
I20260620 21:45:27.255260 3268 ts_tablet_manager.cc:1397] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 21:45:27.255548 3270 tablet_bootstrap.cc:492] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020: No bootstrap required, opened a new log
I20260620 21:45:27.255631 3270 ts_tablet_manager.cc:1397] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260620 21:45:27.255686 3268 raft_consensus.cc:348] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.255808 3268 raft_consensus.cc:374] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:27.255844 3268 raft_consensus.cc:729] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901625bab3e94ae1a56386bfaa752a28, State: Initialized, Role: FOLLOWER
I20260620 21:45:27.255909 3269 tablet_bootstrap.cc:492] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4: No bootstrap required, opened a new log
I20260620 21:45:27.255990 3269 ts_tablet_manager.cc:1397] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 21:45:27.255964 3268 consensus_queue.cc:260] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.256075 3270 raft_consensus.cc:348] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.256201 3270 raft_consensus.cc:374] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:27.256239 3270 raft_consensus.cc:729] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f3ae311f6e4b4fb89997efb47b9020, State: Initialized, Role: FOLLOWER
I20260620 21:45:27.256286 3268 ts_tablet_manager.cc:1428] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:45:27.256357 3270 consensus_queue.cc:260] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.256429 3269 raft_consensus.cc:348] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.256528 3269 raft_consensus.cc:374] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:27.256500 3288 tablet_replica.cc:439] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } }
I20260620 21:45:27.256579 3269 raft_consensus.cc:729] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2125c1d7cdf441da86331d7b1aa2d2d4, State: Initialized, Role: FOLLOWER
I20260620 21:45:27.256625 3288 tablet_replica.cc:442] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:27.256596 3276 tablet_replica.cc:439] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } }
I20260620 21:45:27.256670 3270 ts_tablet_manager.cc:1428] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:45:27.256701 3276 tablet_replica.cc:442] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:27.256688 3269 consensus_queue.cc:260] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.256938 3269 ts_tablet_manager.cc:1428] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:45:27.256918 3275 tablet_replica.cc:439] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } }
I20260620 21:45:27.257031 3275 tablet_replica.cc:442] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:27.282675 3276 raft_consensus.cc:482] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:27.282845 3276 raft_consensus.cc:504] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.283357 3276 leader_election.cc:288] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2125c1d7cdf441da86331d7b1aa2d2d4 (127.1.155.129:41543), 901625bab3e94ae1a56386bfaa752a28 (127.1.155.130:39043)
I20260620 21:45:27.283732 2919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2e716c5c304fb5a70a5174a83b94a5" candidate_uuid: "61f3ae311f6e4b4fb89997efb47b9020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" is_pre_election: true
I20260620 21:45:27.283746 3059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2e716c5c304fb5a70a5174a83b94a5" candidate_uuid: "61f3ae311f6e4b4fb89997efb47b9020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901625bab3e94ae1a56386bfaa752a28" is_pre_election: true
I20260620 21:45:27.283881 2919 raft_consensus.cc:2454] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61f3ae311f6e4b4fb89997efb47b9020 in term 0.
I20260620 21:45:27.283895 3059 raft_consensus.cc:2454] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61f3ae311f6e4b4fb89997efb47b9020 in term 0.
I20260620 21:45:27.284196 3132 leader_election.cc:302] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2125c1d7cdf441da86331d7b1aa2d2d4, 61f3ae311f6e4b4fb89997efb47b9020; no voters:
I20260620 21:45:27.284389 3276 raft_consensus.cc:2781] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:27.284451 3276 raft_consensus.cc:482] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:27.284525 3276 raft_consensus.cc:3037] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:27.285652 3276 raft_consensus.cc:504] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.286036 3276 leader_election.cc:288] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [CANDIDATE]: Term 1 election: Requested vote from peers 2125c1d7cdf441da86331d7b1aa2d2d4 (127.1.155.129:41543), 901625bab3e94ae1a56386bfaa752a28 (127.1.155.130:39043)
I20260620 21:45:27.286341 2919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2e716c5c304fb5a70a5174a83b94a5" candidate_uuid: "61f3ae311f6e4b4fb89997efb47b9020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4"
I20260620 21:45:27.286417 3059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2e716c5c304fb5a70a5174a83b94a5" candidate_uuid: "61f3ae311f6e4b4fb89997efb47b9020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901625bab3e94ae1a56386bfaa752a28"
I20260620 21:45:27.286463 2919 raft_consensus.cc:3037] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:27.286558 3059 raft_consensus.cc:3037] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:27.287325 2919 raft_consensus.cc:2454] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61f3ae311f6e4b4fb89997efb47b9020 in term 1.
I20260620 21:45:27.287523 3059 raft_consensus.cc:2454] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61f3ae311f6e4b4fb89997efb47b9020 in term 1.
I20260620 21:45:27.287588 3132 leader_election.cc:302] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2125c1d7cdf441da86331d7b1aa2d2d4, 61f3ae311f6e4b4fb89997efb47b9020; no voters:
I20260620 21:45:27.287748 3276 raft_consensus.cc:2781] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:27.287879 3276 raft_consensus.cc:686] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [term 1 LEADER]: Becoming Leader. State: Replica: 61f3ae311f6e4b4fb89997efb47b9020, State: Running, Role: LEADER
I20260620 21:45:27.288020 3276 consensus_queue.cc:237] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } }
I20260620 21:45:27.288693 3298 tablet_replica.cc:439] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020: TxnStatusTablet state changed. Reason: New leader 61f3ae311f6e4b4fb89997efb47b9020. Latest consensus state: current_term: 1 leader_uuid: "61f3ae311f6e4b4fb89997efb47b9020" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } }
I20260620 21:45:27.288807 3298 tablet_replica.cc:442] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:45:27.289105 3310 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 21:45:27.289268 3310 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 21:45:27.289750 2760 catalog_manager.cc:5495] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61f3ae311f6e4b4fb89997efb47b9020 (127.1.155.131). New cstate: current_term: 1 leader_uuid: "61f3ae311f6e4b4fb89997efb47b9020" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } health_report { overall_health: HEALTHY } } }
I20260620 21:45:27.310316 3265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:27.306574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.1.155.131:46371)
0620 21:45:27.306706 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:27.306709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:27.306722 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:45:27.306872 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:27.306875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:27.306923 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:27.307001 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:27.307259 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.307738 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.308932 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.309154 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.309233 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:27.309522 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:27.309977 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:27.310084 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:27.310168 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 21:45:27.310210 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:45:27.310374 3305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:27.306461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:46371 (local address 127.0.0.1:60446)
0620 21:45:27.306630 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:27.306645 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:27.306742 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:27.307044 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:27.307047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:27.307055 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:45:27.307163 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:27.307168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.307836 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.307840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:27.308813 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:27.308823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.309257 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.309261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:27.309348 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:27.309397 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:45:27.310067 (+ 670us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:45:27.310081 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 21:45:27.310158 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 21:45:27.310238 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 21:45:27.313184 2919 raft_consensus.cc:1261] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 1 FOLLOWER]: Refusing update from remote peer 61f3ae311f6e4b4fb89997efb47b9020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:27.313405 3059 raft_consensus.cc:1261] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28 [term 1 FOLLOWER]: Refusing update from remote peer 61f3ae311f6e4b4fb89997efb47b9020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:27.313633 3276 consensus_queue.cc:1035] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:27.313884 3298 consensus_queue.cc:1035] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:27.315176 3275 tablet_replica.cc:439] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4: TxnStatusTablet state changed. Reason: New leader 61f3ae311f6e4b4fb89997efb47b9020. Latest consensus state: current_term: 1 leader_uuid: "61f3ae311f6e4b4fb89997efb47b9020" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } }
I20260620 21:45:27.315286 3275 tablet_replica.cc:442] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:27.316257 3288 tablet_replica.cc:439] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28: TxnStatusTablet state changed. Reason: New leader 61f3ae311f6e4b4fb89997efb47b9020. Latest consensus state: current_term: 1 leader_uuid: "61f3ae311f6e4b4fb89997efb47b9020" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } }
I20260620 21:45:27.316380 3288 tablet_replica.cc:442] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:27.317670 3276 tablet_replica.cc:439] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61f3ae311f6e4b4fb89997efb47b9020" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } }
I20260620 21:45:27.317767 3276 tablet_replica.cc:442] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:45:27.319032 3283 mvcc.cc:205] Tried to move back new op lower bound from 7299038934269923328 to 7299038934173155328. Current Snapshot: MvccSnapshot[applied={T|T < 7299038934269923328 or (T in {7299038934269923328})}]
I20260620 21:45:27.319232 3275 tablet_replica.cc:439] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61f3ae311f6e4b4fb89997efb47b9020" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } }
I20260620 21:45:27.319319 3275 tablet_replica.cc:442] T cb2e716c5c304fb5a70a5174a83b94a5 P 2125c1d7cdf441da86331d7b1aa2d2d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:27.319799 3298 tablet_replica.cc:439] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61f3ae311f6e4b4fb89997efb47b9020" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } }
I20260620 21:45:27.319904 3298 tablet_replica.cc:442] T cb2e716c5c304fb5a70a5174a83b94a5 P 61f3ae311f6e4b4fb89997efb47b9020: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:45:27.322674 3288 tablet_replica.cc:439] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61f3ae311f6e4b4fb89997efb47b9020" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 } } peers { permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 } } peers { permanent_uuid: "61f3ae311f6e4b4fb89997efb47b9020" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 46371 } } }
I20260620 21:45:27.322778 3288 tablet_replica.cc:442] T cb2e716c5c304fb5a70a5174a83b94a5 P 901625bab3e94ae1a56386bfaa752a28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:27.325695 3265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:27.322111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60456 (local address 127.1.155.131:46371)
0620 21:45:27.322206 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:27.322210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:27.322340 (+ 130us) server_negotiation.cc:407] Connection header received
0620 21:45:27.322447 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:27.322450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:27.322495 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:27.322576 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:27.323173 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.323909 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.324713 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.324949 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.325054 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:27.325294 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:27.325368 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:27.325444 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:27.325533 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 21:45:27.325574 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 21:45:27.335258 3265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:27.332331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:33525 (local address 127.1.155.131:46371)
0620 21:45:27.332441 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:27.332444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:27.332455 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:45:27.332570 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:27.332572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:27.332608 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:27.332688 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:27.332960 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.333422 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.334307 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.334510 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.334611 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:27.334858 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:27.334929 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:27.335005 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:27.335081 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 21:45:27.335117 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":45}
I20260620 21:45:27.335258 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:27.332264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:46371 (local address 127.1.155.131:33525)
0620 21:45:27.332391 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:27.332405 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:27.332487 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:27.332693 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:27.332696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:27.332702 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:45:27.332858 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:27.332870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.333520 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.333526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:27.334201 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:27.334209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.334609 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.334613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:27.334699 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:27.334741 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:45:27.335006 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:45:27.335012 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:45:27.335074 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 21:45:27.335136 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":43}
I20260620 21:45:27.341512 2919 raft_consensus.cc:1261] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 1 FOLLOWER]: Refusing update from remote peer 61f3ae311f6e4b4fb89997efb47b9020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:27.341512 3059 raft_consensus.cc:1261] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28 [term 1 FOLLOWER]: Refusing update from remote peer 61f3ae311f6e4b4fb89997efb47b9020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:27.341907 3276 consensus_queue.cc:1035] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "901625bab3e94ae1a56386bfaa752a28" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 39043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:27.342037 3298 consensus_queue.cc:1035] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2125c1d7cdf441da86331d7b1aa2d2d4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:27.377710 2761 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:27.377893 2761 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:27.379889 2761 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b24d96881a2f48c585fd18acd951c1f3: Sending DeleteTablet for 3 replicas of tablet 6bea584d39a743b3b3fabd24169faf55
I20260620 21:45:27.380448 2899 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bea584d39a743b3b3fabd24169faf55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:27 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:47856
I20260620 21:45:27.380571 3039 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bea584d39a743b3b3fabd24169faf55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:27 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:46644
I20260620 21:45:27.380669 3292 tablet_replica.cc:330] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4: stopping tablet replica
I20260620 21:45:27.380791 3292 raft_consensus.cc:2229] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:27.380939 3292 raft_consensus.cc:2258] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:27.381106 3293 tablet_replica.cc:330] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28: stopping tablet replica
I20260620 21:45:27.381109 3179 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bea584d39a743b3b3fabd24169faf55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:27 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:60444
I20260620 21:45:27.381223 3293 raft_consensus.cc:2229] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:27.381328 3293 raft_consensus.cc:2258] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:27.382040 3291 tablet_replica.cc:330] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020: stopping tablet replica
I20260620 21:45:27.382148 3291 raft_consensus.cc:2229] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:45:27.382398 3291 raft_consensus.cc:2258] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:27.382499 3293 ts_tablet_manager.cc:1905] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:27.383006 3292 ts_tablet_manager.cc:1905] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:27.385104 3293 ts_tablet_manager.cc:1918] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:45:27.385176 3293 log.cc:1196] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/6bea584d39a743b3b3fabd24169faf55
I20260620 21:45:27.385429 3293 ts_tablet_manager.cc:1939] T 6bea584d39a743b3b3fabd24169faf55 P 901625bab3e94ae1a56386bfaa752a28: Deleting consensus metadata
I20260620 21:45:27.385609 3292 ts_tablet_manager.cc:1918] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:45:27.385690 3292 log.cc:1196] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-0/wal/wals/6bea584d39a743b3b3fabd24169faf55
I20260620 21:45:27.385954 3292 ts_tablet_manager.cc:1939] T 6bea584d39a743b3b3fabd24169faf55 P 2125c1d7cdf441da86331d7b1aa2d2d4: Deleting consensus metadata
I20260620 21:45:27.386215 2746 catalog_manager.cc:4841] TS 901625bab3e94ae1a56386bfaa752a28 (127.1.155.130:39043): tablet 6bea584d39a743b3b3fabd24169faf55 (table test-table [id=38a162237f994bc2aa94c75f699b83f4]) successfully deleted
I20260620 21:45:27.386569 2748 catalog_manager.cc:4841] TS 2125c1d7cdf441da86331d7b1aa2d2d4 (127.1.155.129:41543): tablet 6bea584d39a743b3b3fabd24169faf55 (table test-table [id=38a162237f994bc2aa94c75f699b83f4]) successfully deleted
I20260620 21:45:27.387003 3291 ts_tablet_manager.cc:1905] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:27.389494 3291 ts_tablet_manager.cc:1918] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:45:27.389570 3291 log.cc:1196] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/6bea584d39a743b3b3fabd24169faf55
I20260620 21:45:27.389808 3291 ts_tablet_manager.cc:1939] T 6bea584d39a743b3b3fabd24169faf55 P 61f3ae311f6e4b4fb89997efb47b9020: Deleting consensus metadata
I20260620 21:45:27.390508 2749 catalog_manager.cc:4841] TS 61f3ae311f6e4b4fb89997efb47b9020 (127.1.155.131:46371): tablet 6bea584d39a743b3b3fabd24169faf55 (table test-table [id=38a162237f994bc2aa94c75f699b83f4]) successfully deleted
Jun 20 21:45:27 dist-test-slave-5bvn krb5kdc[2717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.155.129@KRBTEST.COM
I20260620 21:45:27.402343 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:27.390641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47866 (local address 127.1.155.129:41543)
0620 21:45:27.390758 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:27.390762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:27.391315 (+ 553us) server_negotiation.cc:407] Connection header received
0620 21:45:27.391431 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:27.391434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:27.391483 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:27.391569 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:27.392367 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.393142 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.394313 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.394653 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.394757 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:27.398567 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:27.398576 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:27.398590 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:27.398615 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:27.400501 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:27.400965 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:27.400971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:27.400973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:27.401066 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:27.401331 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:27.401334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:27.401335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:27.401601 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:27.401779 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:27.402075 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 21:45:27.402191 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 21:45:27.403065 2959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47866
I20260620 21:45:27.412835 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:27.404746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.1.155.190:42155)
0620 21:45:27.404860 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:27.404865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:27.405253 (+ 388us) server_negotiation.cc:407] Connection header received
0620 21:45:27.405344 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:27.405347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:27.405396 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:27.405482 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:27.406286 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.406791 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.407800 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:27.407999 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:27.408105 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:27.409543 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:27.409546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:27.409548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:27.409569 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:27.411002 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:27.411488 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:27.411491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:27.411493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:27.411563 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:27.411872 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:27.411875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:27.411878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:27.412084 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:27.412180 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:27.412447 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 21:45:27.412594 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 21:45:27.413256 2761 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58732
I20260620 21:45:27.414945 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 2834
I20260620 21:45:27.423552 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 2974
I20260620 21:45:27.431085 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 3114
I20260620 21:45:27.437690 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 2730
2026-06-20T21:45:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2763 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:45:27 dist-test-slave-5bvn krb5kdc[3328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:45:27 dist-test-slave-5bvn krb5kdc[3328](info): set up 2 sockets
Jun 20 21:45:27 dist-test-slave-5bvn krb5kdc[3328](info): commencing operation
krb5kdc: starting...
W20260620 21:45:28.481647 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:45:28 dist-test-slave-5bvn krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:45:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:45:28.601497 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:38851
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:41235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:38851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:28.710114 3341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:28.710419 3341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:28.710500 3341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:28.714576 3341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:28.714661 3341 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 21:45:28.714694 3341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:28.714763 3341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:28.714802 3341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:28.714848 3341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:45:28.720135 3341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:41235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:38851
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:38851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:28.721223 3341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:28.722292 3341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:28.730747 3346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:28.730746 3349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:28.730754 3347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:28.731029 3341 server_base.cc:1034] running on GCE node
I20260620 21:45:28.731627 3341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:28.732748 3341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:28.733935 3341 hybrid_clock.cc:648] HybridClock initialized: now 1781991928733904 us; error 56 us; skew 500 ppm
Jun 20 21:45:28 dist-test-slave-5bvn krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991928, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:28.736835 3341 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:45:28.737689 3341 webserver.cc:460] Webserver started at http://127.1.155.190:41471/ using document root <none> and password file <none>
I20260620 21:45:28.738292 3341 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:28.738386 3341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:28.738615 3341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:28.740609 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "3eb34716a2ea4937986b9eb80c8f86ac"
format_stamp: "Formatted at 2026-06-20 21:45:28 on dist-test-slave-5bvn"
server_key: "9a7a50587cc0d5f5779254ccf1c51e5c"
server_key_iv: "90919f7aefbefd811bf992f19e70f2cb"
server_key_version: "encryptionkey@0"
I20260620 21:45:28.741170 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "3eb34716a2ea4937986b9eb80c8f86ac"
format_stamp: "Formatted at 2026-06-20 21:45:28 on dist-test-slave-5bvn"
server_key: "9a7a50587cc0d5f5779254ccf1c51e5c"
server_key_iv: "90919f7aefbefd811bf992f19e70f2cb"
server_key_version: "encryptionkey@0"
I20260620 21:45:28.744863 3341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:45:28.747370 3356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:28.748579 3341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 21:45:28.748752 3341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "3eb34716a2ea4937986b9eb80c8f86ac"
format_stamp: "Formatted at 2026-06-20 21:45:28 on dist-test-slave-5bvn"
server_key: "9a7a50587cc0d5f5779254ccf1c51e5c"
server_key_iv: "90919f7aefbefd811bf992f19e70f2cb"
server_key_version: "encryptionkey@0"
I20260620 21:45:28.748871 3341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:28.763443 3341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:28.766839 3341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:28.767043 3341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:28.774919 3341 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:38851
I20260620 21:45:28.774997 3418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:38851 every 8 connection(s)
I20260620 21:45:28.775876 3341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:28.777496 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 3341
I20260620 21:45:28.777635 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:28.777966 1646 external_mini_cluster.cc:1428] Setting key b0507a7256eaffdf5db87ee6dbef3476
I20260620 21:45:28.778837 3419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:28.784443 3419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac: Bootstrap starting.
Jun 20 21:45:28 dist-test-slave-5bvn krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:28.786962 3419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:28.787674 3419 log.cc:824] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:28.789691 3419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac: No bootstrap required, opened a new log
I20260620 21:45:28.792780 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:28.779324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33234 (local address 127.1.155.190:38851)
0620 21:45:28.779772 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:28.779788 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:28.779825 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:45:28.780477 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:28.780492 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:28.780811 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:28.781183 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:28.782013 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:28.783045 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:28.783946 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:28.784455 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:28.784630 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:28.787453 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:28.787459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:28.787472 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:28.787501 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:28.789943 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:28.790461 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:28.790466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:28.790471 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:28.790585 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:28.790869 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:28.790873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:28.790877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:28.791263 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:28.791469 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:28.791756 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 21:45:28.792058 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":145,"threads_started":1}
I20260620 21:45:28.792935 3419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb34716a2ea4937986b9eb80c8f86ac" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38851 } }
I20260620 21:45:28.793217 3419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:28.793277 3419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb34716a2ea4937986b9eb80c8f86ac, State: Initialized, Role: FOLLOWER
I20260620 21:45:28.793720 3419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb34716a2ea4937986b9eb80c8f86ac" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38851 } }
I20260620 21:45:28.793921 3419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:28.794020 3419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:28.794119 3419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:28.795050 3419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb34716a2ea4937986b9eb80c8f86ac" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38851 } }
I20260620 21:45:28.795375 3419 leader_election.cc:302] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3eb34716a2ea4937986b9eb80c8f86ac; no voters:
I20260620 21:45:28.795627 3419 leader_election.cc:288] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:28.795758 3424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:28.795974 3424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [term 1 LEADER]: Becoming Leader. State: Replica: 3eb34716a2ea4937986b9eb80c8f86ac, State: Running, Role: LEADER
I20260620 21:45:28.796298 3424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb34716a2ea4937986b9eb80c8f86ac" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38851 } }
I20260620 21:45:28.796849 3419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:28.797940 3426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eb34716a2ea4937986b9eb80c8f86ac. Latest consensus state: current_term: 1 leader_uuid: "3eb34716a2ea4937986b9eb80c8f86ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb34716a2ea4937986b9eb80c8f86ac" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38851 } } }
I20260620 21:45:28.798079 3426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:28.798369 3425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3eb34716a2ea4937986b9eb80c8f86ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb34716a2ea4937986b9eb80c8f86ac" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38851 } } }
I20260620 21:45:28.798501 3433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:28.798517 3425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:28.801441 3433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:28.806914 3433 catalog_manager.cc:1347] Generated new cluster ID: 3140001500c14d13b9406e6c2b256d2e
I20260620 21:45:28.807008 3433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:28.824124 3433 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:28.825131 3433 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:28.833585 3433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac: Generated new TSK 0
I20260620 21:45:28.834429 3433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:45:28.865356 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:38851
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:41235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:28.979552 3445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:28.979864 3445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:28.979915 3445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:28.984169 3445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:28.984239 3445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:28.984282 3445 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 21:45:28.984297 3445 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:45:28.984329 3445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:45:28.989794 3445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:41235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.155.190:38851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:28.990949 3445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:28.991923 3445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:29.001533 3450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:29.001535 3453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:29.001528 3451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:29.001937 3445 server_base.cc:1034] running on GCE node
I20260620 21:45:29.002341 3445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:29.002969 3445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:29.004163 3445 hybrid_clock.cc:648] HybridClock initialized: now 1781991929004148 us; error 34 us; skew 500 ppm
Jun 20 21:45:29 dist-test-slave-5bvn krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991929, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:29.006980 3445 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:45:29.007819 3445 webserver.cc:460] Webserver started at http://127.1.155.129:35633/ using document root <none> and password file <none>
I20260620 21:45:29.008340 3445 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:29.008397 3445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:29.008559 3445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:29.010931 3445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "b67e25d810534513892c093fa33600a6"
format_stamp: "Formatted at 2026-06-20 21:45:29 on dist-test-slave-5bvn"
server_key: "3329920bc18831a8a4d59457410258cf"
server_key_iv: "221d56e986cd35545c76e4f410b51ce2"
server_key_version: "encryptionkey@0"
I20260620 21:45:29.011487 3445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "b67e25d810534513892c093fa33600a6"
format_stamp: "Formatted at 2026-06-20 21:45:29 on dist-test-slave-5bvn"
server_key: "3329920bc18831a8a4d59457410258cf"
server_key_iv: "221d56e986cd35545c76e4f410b51ce2"
server_key_version: "encryptionkey@0"
I20260620 21:45:29.015293 3445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:45:29.017804 3460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:29.019550 3445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:45:29.019659 3445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "b67e25d810534513892c093fa33600a6"
format_stamp: "Formatted at 2026-06-20 21:45:29 on dist-test-slave-5bvn"
server_key: "3329920bc18831a8a4d59457410258cf"
server_key_iv: "221d56e986cd35545c76e4f410b51ce2"
server_key_version: "encryptionkey@0"
I20260620 21:45:29.019819 3445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:29.037346 3445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:29.040545 3445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:29.040746 3445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:29.042414 3445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:29.042475 3445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:29.042539 3445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:29.042572 3445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:45:29 dist-test-slave-5bvn krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991929, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:29.053498 3445 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:42235
I20260620 21:45:29.053592 3575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:42235 every 8 connection(s)
I20260620 21:45:29.054555 3445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:45:29.056602 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.043205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:48557 (local address 127.1.155.190:38851)
0620 21:45:29.043336 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:29.043339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:29.044321 (+ 982us) server_negotiation.cc:407] Connection header received
0620 21:45:29.045326 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:29.045329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:29.045387 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:29.045471 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:29.047172 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.047708 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.049076 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.049318 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.049438 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.052762 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:29.052766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:29.052769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:29.052806 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:29.054472 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.055029 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.055033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.055035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.055121 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.055548 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.055551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.055552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.055746 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:29.055841 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:29.056369 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 21:45:29.056482 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:45:29.057317 3469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.043716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:38851 (local address 127.1.155.129:48557)
0620 21:45:29.044179 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:29.044219 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:29.045120 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:29.045760 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:29.045769 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:29.046313 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:29.046997 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.047019 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.047844 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.047848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.048938 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.048947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.049490 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.049494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.049676 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.050323 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:29.050346 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:29.052592 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:29.054617 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:29.054623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:29.054637 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:29.054922 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:29.055251 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:29.055255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:29.055257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:29.055426 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:29.055893 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:29.055898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:29.056120 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 21:45:29.056285 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 21:45:29.056531 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":134,"threads_started":1}
I20260620 21:45:29.058609 3576 heartbeater.cc:347] Connected to a master server at 127.1.155.190:38851
I20260620 21:45:29.058739 3576 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:29.059315 3576 heartbeater.cc:511] Master 127.1.155.190:38851 requested a full tablet report, sending...
I20260620 21:45:29.061041 3372 ts_manager.cc:194] Registered new tserver with Master: b67e25d810534513892c093fa33600a6 (127.1.155.129:42235)
I20260620 21:45:29.062103 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 3445
I20260620 21:45:29.062217 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:45:29.062507 1646 external_mini_cluster.cc:1428] Setting key 1903b821eba21b828effbe7d6b2872e5
I20260620 21:45:29.062580 3372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:48557
I20260620 21:45:29.069211 3582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.060807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:38851 (local address 127.1.155.129:54079)
0620 21:45:29.061126 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:29.061146 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:29.061296 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:29.061555 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:29.061558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:29.061875 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:29.062006 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.062014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.063001 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.063005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.063984 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.063994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.064746 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.064751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.064918 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.065484 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:29.065503 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:29.065958 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:29.067830 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:29.067833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:29.067834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:29.068014 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:29.068311 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:29.068312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:29.068314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:29.068382 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:29.068822 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:29.068824 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:29.068895 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 21:45:29.068976 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 21:45:29.069077 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":122,"threads_started":1}
I20260620 21:45:29.069419 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.060949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:54079 (local address 127.1.155.190:38851)
0620 21:45:29.061088 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:29.061091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:29.061223 (+ 132us) server_negotiation.cc:407] Connection header received
0620 21:45:29.061402 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:29.061405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:29.061442 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:29.061508 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:29.062137 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.062847 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.064363 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.064604 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.064735 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.066110 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:29.066113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:29.066115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:29.066135 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:29.067715 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.068111 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.068117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.068122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.068214 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.068472 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.068478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.068481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.068726 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:29.068836 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:29.069063 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 21:45:29.069191 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:45:29.091229 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:38851
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:41235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:29.202327 3585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:29.202569 3585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:29.202615 3585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:29.206612 3585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:29.206661 3585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:29.206701 3585 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 21:45:29.206715 3585 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:45:29.206748 3585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:45:29.211827 3585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:41235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.155.190:38851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:29.212745 3585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:29.213649 3585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:29.223197 3591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:29.223337 3585 server_base.cc:1034] running on GCE node
W20260620 21:45:29.223464 3590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:29.223173 3593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:29.223879 3585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:29.224538 3585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:29.225736 3585 hybrid_clock.cc:648] HybridClock initialized: now 1781991929225717 us; error 29 us; skew 500 ppm
Jun 20 21:45:29 dist-test-slave-5bvn krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991929, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:29.228386 3585 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:45:29.229207 3585 webserver.cc:460] Webserver started at http://127.1.155.130:39107/ using document root <none> and password file <none>
I20260620 21:45:29.229754 3585 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:29.229847 3585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:29.230093 3585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:29.232038 3585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "b4e8be01536a49b1a033922d9c8c7e69"
format_stamp: "Formatted at 2026-06-20 21:45:29 on dist-test-slave-5bvn"
server_key: "74da4cd976c622a1cd308a80f3983ad9"
server_key_iv: "ed31665e066fbef016d9fadd82121eb2"
server_key_version: "encryptionkey@0"
I20260620 21:45:29.232586 3585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "b4e8be01536a49b1a033922d9c8c7e69"
format_stamp: "Formatted at 2026-06-20 21:45:29 on dist-test-slave-5bvn"
server_key: "74da4cd976c622a1cd308a80f3983ad9"
server_key_iv: "ed31665e066fbef016d9fadd82121eb2"
server_key_version: "encryptionkey@0"
I20260620 21:45:29.236202 3585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 21:45:29.238620 3600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:29.239840 3585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:45:29.239980 3585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "b4e8be01536a49b1a033922d9c8c7e69"
format_stamp: "Formatted at 2026-06-20 21:45:29 on dist-test-slave-5bvn"
server_key: "74da4cd976c622a1cd308a80f3983ad9"
server_key_iv: "ed31665e066fbef016d9fadd82121eb2"
server_key_version: "encryptionkey@0"
I20260620 21:45:29.240100 3585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:29.264832 3585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:29.267782 3585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:29.267984 3585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:29.269439 3585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:29.269521 3585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:29.269593 3585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:29.269635 3585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:45:29 dist-test-slave-5bvn krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991929, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:29.280297 3585 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:43753
I20260620 21:45:29.280541 3715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:43753 every 8 connection(s)
I20260620 21:45:29.281279 3585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:45:29.283707 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.270234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:38097 (local address 127.1.155.190:38851)
0620 21:45:29.270386 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:29.270391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:29.271346 (+ 955us) server_negotiation.cc:407] Connection header received
0620 21:45:29.272607 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:29.272614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:29.272681 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:29.272841 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:29.274502 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.275402 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.276538 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.276765 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.276892 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.279574 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:29.279578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:29.279581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:29.279612 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:29.281559 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.282286 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.282289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.282290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.282372 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.282742 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.282744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.282745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.282917 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:29.282994 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:29.283496 (+ 502us) server_negotiation.cc:299] Negotiation successful
0620 21:45:29.283593 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:45:29.284502 3609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.270680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:38851 (local address 127.1.155.130:38097)
0620 21:45:29.271165 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:29.271225 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:29.272361 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:29.273051 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:29.273062 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:29.273537 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:29.274302 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.274325 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.275538 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.275542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.276404 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.276413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.276935 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.276939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.277124 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.277754 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:29.277770 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:29.279406 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:29.281702 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:29.281709 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:29.281725 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:29.282170 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:29.282467 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:29.282469 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:29.282471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:29.282629 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:29.283032 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:29.283037 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:29.283274 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 21:45:29.283440 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 21:45:29.283705 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":107,"threads_started":1}
I20260620 21:45:29.285669 3716 heartbeater.cc:347] Connected to a master server at 127.1.155.190:38851
I20260620 21:45:29.285852 3716 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:29.286381 3716 heartbeater.cc:511] Master 127.1.155.190:38851 requested a full tablet report, sending...
I20260620 21:45:29.287449 3373 ts_manager.cc:194] Registered new tserver with Master: b4e8be01536a49b1a033922d9c8c7e69 (127.1.155.130:43753)
I20260620 21:45:29.287755 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 3585
I20260620 21:45:29.287863 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:45:29.288146 1646 external_mini_cluster.cc:1428] Setting key 5ef066f35cec088be71aa0aad9b210f3
I20260620 21:45:29.288270 3373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:38097
I20260620 21:45:29.295801 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.287865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:56107 (local address 127.1.155.190:38851)
0620 21:45:29.287974 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:29.287977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:29.288198 (+ 221us) server_negotiation.cc:407] Connection header received
0620 21:45:29.288315 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:29.288320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:29.288361 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:29.288446 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:29.289318 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.289936 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.290918 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.291160 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.291260 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.292622 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:29.292626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:29.292629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:29.292651 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:29.294160 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.294606 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.294615 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.294617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.294687 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.294962 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.294967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.294969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.295170 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:29.295294 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:29.295545 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 21:45:29.295647 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 21:45:29.295827 3722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.287839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:38851 (local address 127.1.155.130:56107)
0620 21:45:29.288115 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:29.288126 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:29.288235 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:29.288458 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:29.288872 (+ 414us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:29.289072 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:29.289205 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.289212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.290060 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.290063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.290799 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.290808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.291386 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.291389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.291459 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.292010 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:29.292032 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:29.292487 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:29.294278 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:29.294284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:29.294287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:29.294497 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:29.294780 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:29.294784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:29.294786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:29.294872 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:29.295282 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:29.295285 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:29.295383 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 21:45:29.295496 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 21:45:29.295642 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":2,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:45:29.316736 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:38851
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:41235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:29.427198 3725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:29.427436 3725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:29.427513 3725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:29.431505 3725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:29.431557 3725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:29.431624 3725 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 21:45:29.431653 3725 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:45:29.431761 3725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:45:29.436923 3725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:41235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.155.190:38851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:29.437966 3725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:29.438947 3725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:29.448251 3730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:29.448307 3731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:29.448253 3733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:29.448573 3725 server_base.cc:1034] running on GCE node
I20260620 21:45:29.449000 3725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:29.449556 3725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:29.450734 3725 hybrid_clock.cc:648] HybridClock initialized: now 1781991929450662 us; error 63 us; skew 500 ppm
Jun 20 21:45:29 dist-test-slave-5bvn krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991929, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:29.453575 3725 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:45:29.454452 3725 webserver.cc:460] Webserver started at http://127.1.155.131:46155/ using document root <none> and password file <none>
I20260620 21:45:29.455000 3725 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:29.455089 3725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:29.455313 3725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:29.457245 3725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "e35c8149faea4ec8a7c3be229df0055c"
format_stamp: "Formatted at 2026-06-20 21:45:29 on dist-test-slave-5bvn"
server_key: "ae566a0f132d39a2be76ed9378c2182d"
server_key_iv: "c9acab8526d1bfacf6f505270d21ab15"
server_key_version: "encryptionkey@0"
I20260620 21:45:29.457789 3725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "e35c8149faea4ec8a7c3be229df0055c"
format_stamp: "Formatted at 2026-06-20 21:45:29 on dist-test-slave-5bvn"
server_key: "ae566a0f132d39a2be76ed9378c2182d"
server_key_iv: "c9acab8526d1bfacf6f505270d21ab15"
server_key_version: "encryptionkey@0"
I20260620 21:45:29.461447 3725 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:45:29.463937 3740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:29.465144 3725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:45:29.465274 3725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "e35c8149faea4ec8a7c3be229df0055c"
format_stamp: "Formatted at 2026-06-20 21:45:29 on dist-test-slave-5bvn"
server_key: "ae566a0f132d39a2be76ed9378c2182d"
server_key_iv: "c9acab8526d1bfacf6f505270d21ab15"
server_key_version: "encryptionkey@0"
I20260620 21:45:29.465380 3725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:29.481596 3725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:29.484930 3725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:29.485183 3725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:29.486814 3725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:29.486891 3725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:29.486959 3725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:29.487010 3725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:45:29 dist-test-slave-5bvn krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991929, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:29.497789 3725 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:33277
I20260620 21:45:29.498060 3855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:33277 every 8 connection(s)
I20260620 21:45:29.498849 3725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:45:29.500674 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.487707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:45459 (local address 127.1.155.190:38851)
0620 21:45:29.487860 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:29.487865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:29.488769 (+ 904us) server_negotiation.cc:407] Connection header received
0620 21:45:29.489904 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:29.489907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:29.489961 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:29.490047 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:29.491324 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.492090 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.493534 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.493823 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.493986 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.496843 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:29.496848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:29.496851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:29.496881 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:29.498611 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.499209 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.499215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.499217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.499314 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.499716 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.499719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.499720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.499897 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:29.499987 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:29.500463 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 21:45:29.500560 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:45:29.501466 3749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.488126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:38851 (local address 127.1.155.131:45459)
0620 21:45:29.488585 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:29.488634 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:29.489627 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:29.490246 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:29.490253 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:29.490590 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:29.491178 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.491192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.492214 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.492220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.493416 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.493425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.494035 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.494038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.494211 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.494842 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:29.494858 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:29.496621 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:29.498749 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:29.498755 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:29.498769 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:29.499097 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:29.499419 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:29.499422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:29.499424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:29.499592 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:29.500020 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:29.500025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:29.500252 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 21:45:29.500412 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 21:45:29.500653 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":133,"threads_started":1}
I20260620 21:45:29.502694 3856 heartbeater.cc:347] Connected to a master server at 127.1.155.190:38851
I20260620 21:45:29.502854 3856 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:29.503070 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 3725
I20260620 21:45:29.503201 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:45:29.503427 3856 heartbeater.cc:511] Master 127.1.155.190:38851 requested a full tablet report, sending...
I20260620 21:45:29.503561 1646 external_mini_cluster.cc:1428] Setting key 847c402539071388945cc7b952e83207
I20260620 21:45:29.504467 3372 ts_manager.cc:194] Registered new tserver with Master: e35c8149faea4ec8a7c3be229df0055c (127.1.155.131:33277)
I20260620 21:45:29.505232 3372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:45459
I20260620 21:45:29.505489 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:45:29.512686 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.505195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:46237 (local address 127.1.155.190:38851)
0620 21:45:29.505297 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:29.505301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:29.505502 (+ 201us) server_negotiation.cc:407] Connection header received
0620 21:45:29.505646 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:29.505649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:29.505696 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:29.505766 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:29.506283 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.506778 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.507705 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.507910 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.507987 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.509069 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:29.509073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:29.509075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:29.509095 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:29.510888 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.511416 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.511419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.511420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.511491 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.511802 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.511828 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.511829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.511979 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:29.512053 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:29.512362 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 21:45:29.512482 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 21:45:29.512760 3862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.504864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:38851 (local address 127.1.155.131:46237)
0620 21:45:29.505161 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:29.505177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:29.505546 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:29.505799 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:29.505802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:29.506053 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:29.506167 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.506173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.506904 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.506907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.507598 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.507609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.508018 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.508025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.508138 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.508623 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:29.508642 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:29.508950 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:29.511019 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:29.511021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:29.511023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:29.511310 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:29.511589 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:29.511595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:29.511600 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:29.511709 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:29.512072 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:29.512077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:29.512195 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 21:45:29.512312 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 21:45:29.512475 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":7,"thread_start_us":91,"threads_started":1}
Jun 20 21:45:29 dist-test-slave-5bvn krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:45:29 dist-test-slave-5bvn krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:29.532863 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.524293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33250 (local address 127.1.155.190:38851)
0620 21:45:29.524417 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:29.524421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:29.524434 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:45:29.524474 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:29.524476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:29.524513 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:29.524582 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:29.525291 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.525799 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.527052 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.527298 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.527400 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.529802 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:29.529805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:29.529808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:29.529923 (+ 115us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:29.531274 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.531683 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.531686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.531687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.531767 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.532060 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.532063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.532064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.532285 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:29.532416 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:29.532628 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 21:45:29.532721 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:45:29.535730 3372 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:45:29.538662 3372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:29.553246 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.547084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34656 (local address 127.1.155.131:33277)
0620 21:45:29.548320 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:29.548337 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:29.548357 (+ 20us) server_negotiation.cc:407] Connection header received
0620 21:45:29.548428 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:29.548436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:29.548632 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:29.549159 (+ 527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:29.549301 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.550110 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.551687 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.552515 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.552623 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.552763 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:29.552998 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 21:45:29.553095 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1144,"thread_start_us":101,"threads_started":1}
I20260620 21:45:29.553234 3873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.546982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:33277 (local address 127.0.0.1:34656)
0620 21:45:29.547547 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:29.547595 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:29.547709 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:29.548816 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:29.548819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:29.548837 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:29.549011 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.549018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.550242 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.550245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.551517 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.551527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.552622 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.552624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.552745 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.552790 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 21:45:29.552965 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 21:45:29.553086 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":4,"thread_start_us":113,"threads_started":1}
I20260620 21:45:29.553913 3874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.547690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:42235 (local address 127.0.0.1:33890)
0620 21:45:29.548340 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:29.548352 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:29.548444 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:29.548958 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:29.548962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:29.548976 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:29.549110 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.549117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.550454 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.550457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.551955 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.551965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.553252 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.553256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.553570 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.553596 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 21:45:29.553670 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 21:45:29.553741 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":109,"thread_start_us":99,"threads_started":1}
I20260620 21:45:29.554303 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.547048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:43753 (local address 127.0.0.1:48822)
0620 21:45:29.547976 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:29.547990 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:29.548069 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:29.548759 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:29.548764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:29.548786 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:29.548950 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.548958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.550446 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.550449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.551955 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.551965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.553908 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.553912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.554009 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.554032 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:45:29.554100 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 21:45:29.554165 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":103,"thread_start_us":125,"threads_started":1}
I20260620 21:45:29.554371 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.548039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.1.155.129:42235)
0620 21:45:29.548320 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:29.548336 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:29.548417 (+ 81us) server_negotiation.cc:407] Connection header received
0620 21:45:29.548537 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:29.548541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:29.548694 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:29.548834 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:29.549222 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.550346 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.552071 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.552926 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.553069 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.553193 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:29.554134 (+ 941us) server_negotiation.cc:299] Negotiation successful
0620 21:45:29.554221 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":67,"threads_started":1}
I20260620 21:45:29.554702 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.547888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.1.155.130:43753)
0620 21:45:29.548273 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:29.548284 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:29.548308 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:45:29.548390 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:29.548397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:29.548604 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:29.548779 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:29.549067 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.550304 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.552839 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.553624 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.553715 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.553851 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:29.554466 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 21:45:29.554560 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":99,"threads_started":1}
I20260620 21:45:29.556577 3790 tablet_service.cc:1467] Processing CreateTablet for tablet 760f3c6a6c044acbbb804960788982b1 (DEFAULT_TABLE table=test-table [id=17d71033a7b54c84be33cd9b15fe8d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:29.556979 3510 tablet_service.cc:1467] Processing CreateTablet for tablet 760f3c6a6c044acbbb804960788982b1 (DEFAULT_TABLE table=test-table [id=17d71033a7b54c84be33cd9b15fe8d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:29.556977 3641 tablet_service.cc:1467] Processing CreateTablet for tablet 760f3c6a6c044acbbb804960788982b1 (DEFAULT_TABLE table=test-table [id=17d71033a7b54c84be33cd9b15fe8d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:29.557605 3790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 760f3c6a6c044acbbb804960788982b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:29.557962 3641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 760f3c6a6c044acbbb804960788982b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:29.557962 3510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 760f3c6a6c044acbbb804960788982b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:29.564448 3879 tablet_bootstrap.cc:492] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c: Bootstrap starting.
I20260620 21:45:29.565769 3880 tablet_bootstrap.cc:492] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6: Bootstrap starting.
I20260620 21:45:29.565855 3881 tablet_bootstrap.cc:492] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69: Bootstrap starting.
I20260620 21:45:29.566614 3879 tablet_bootstrap.cc:654] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:29.567351 3879 log.cc:824] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:29.567914 3881 tablet_bootstrap.cc:654] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:29.568403 3880 tablet_bootstrap.cc:654] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:29.568881 3881 log.cc:824] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:29.569320 3880 log.cc:824] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:29.569577 3879 tablet_bootstrap.cc:492] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c: No bootstrap required, opened a new log
I20260620 21:45:29.569764 3879 ts_tablet_manager.cc:1397] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:45:29.570885 3881 tablet_bootstrap.cc:492] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69: No bootstrap required, opened a new log
I20260620 21:45:29.571096 3881 ts_tablet_manager.cc:1397] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 21:45:29.571245 3880 tablet_bootstrap.cc:492] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6: No bootstrap required, opened a new log
I20260620 21:45:29.571421 3880 ts_tablet_manager.cc:1397] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 21:45:29.573696 3879 raft_consensus.cc:348] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } }
I20260620 21:45:29.574002 3879 raft_consensus.cc:374] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:29.574085 3879 raft_consensus.cc:729] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35c8149faea4ec8a7c3be229df0055c, State: Initialized, Role: FOLLOWER
I20260620 21:45:29.574139 3881 raft_consensus.cc:348] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } }
I20260620 21:45:29.574446 3881 raft_consensus.cc:374] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:29.574513 3881 raft_consensus.cc:729] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e8be01536a49b1a033922d9c8c7e69, State: Initialized, Role: FOLLOWER
I20260620 21:45:29.574596 3879 consensus_queue.cc:260] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } }
I20260620 21:45:29.574954 3881 consensus_queue.cc:260] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } }
I20260620 21:45:29.575341 3856 heartbeater.cc:503] Master 127.1.155.190:38851 was elected leader, sending a full tablet report...
I20260620 21:45:29.575642 3879 ts_tablet_manager.cc:1428] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:45:29.575903 3881 ts_tablet_manager.cc:1428] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 21:45:29.575969 3880 raft_consensus.cc:348] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } }
I20260620 21:45:29.576200 3880 raft_consensus.cc:374] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:29.576231 3716 heartbeater.cc:503] Master 127.1.155.190:38851 was elected leader, sending a full tablet report...
I20260620 21:45:29.576248 3880 raft_consensus.cc:729] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67e25d810534513892c093fa33600a6, State: Initialized, Role: FOLLOWER
I20260620 21:45:29.576656 3880 consensus_queue.cc:260] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } }
I20260620 21:45:29.577405 3880 ts_tablet_manager.cc:1428] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260620 21:45:29.577734 3576 heartbeater.cc:503] Master 127.1.155.190:38851 was elected leader, sending a full tablet report...
I20260620 21:45:29.674535 3886 raft_consensus.cc:482] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:29.674822 3886 raft_consensus.cc:504] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } }
I20260620 21:45:29.675993 3886 leader_election.cc:288] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e35c8149faea4ec8a7c3be229df0055c (127.1.155.131:33277), b67e25d810534513892c093fa33600a6 (127.1.155.129:42235)
I20260620 21:45:29.680034 3609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.676298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:42235 (local address 127.1.155.130:40361)
0620 21:45:29.676418 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:29.676434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:29.676549 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:29.676833 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:29.676838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:29.676860 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:29.677016 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.677021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.677894 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.677897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.678980 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.678989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.679699 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.679702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.679786 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.679807 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:45:29.679867 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 21:45:29.679923 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":24,"spinlock_wait_cycles":4224}
I20260620 21:45:29.680097 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.676399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:40361 (local address 127.1.155.129:42235)
0620 21:45:29.676522 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:29.676529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:29.676550 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:45:29.676643 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:29.676646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:29.676701 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:29.676782 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:29.677164 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.677743 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.679116 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.679582 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.679754 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.679807 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:29.679899 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 21:45:29.679971 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:45:29.680683 3888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.676302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:33277 (local address 127.1.155.130:38931)
0620 21:45:29.676788 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:29.676803 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:29.676890 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:29.677116 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:29.677121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:29.677159 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:29.677591 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.677596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.678383 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.678386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.679373 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.679380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.680311 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.680316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.680417 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.680445 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:45:29.680524 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 21:45:29.680575 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":113,"spinlock_wait_cycles":768,"thread_start_us":139,"threads_started":1}
I20260620 21:45:29.680747 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.676361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:38931 (local address 127.1.155.131:33277)
0620 21:45:29.676478 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:29.676485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:29.676898 (+ 413us) server_negotiation.cc:407] Connection header received
0620 21:45:29.676980 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:29.676982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:29.677018 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:29.677075 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:29.677715 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.678270 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.679487 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.680062 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.680183 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.680245 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:29.680570 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 21:45:29.680644 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:45:29.680776 3530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760f3c6a6c044acbbb804960788982b1" candidate_uuid: "b4e8be01536a49b1a033922d9c8c7e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67e25d810534513892c093fa33600a6" is_pre_election: true
I20260620 21:45:29.681100 3530 raft_consensus.cc:2454] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4e8be01536a49b1a033922d9c8c7e69 in term 0.
I20260620 21:45:29.681348 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760f3c6a6c044acbbb804960788982b1" candidate_uuid: "b4e8be01536a49b1a033922d9c8c7e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35c8149faea4ec8a7c3be229df0055c" is_pre_election: true
I20260620 21:45:29.681648 3810 raft_consensus.cc:2454] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4e8be01536a49b1a033922d9c8c7e69 in term 0.
I20260620 21:45:29.681665 3602 leader_election.cc:302] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e8be01536a49b1a033922d9c8c7e69, b67e25d810534513892c093fa33600a6; no voters:
I20260620 21:45:29.682014 3886 raft_consensus.cc:2781] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:29.682109 3886 raft_consensus.cc:482] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:29.682163 3886 raft_consensus.cc:3037] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:29.683152 3886 raft_consensus.cc:504] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } }
I20260620 21:45:29.683516 3886 leader_election.cc:288] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [CANDIDATE]: Term 1 election: Requested vote from peers e35c8149faea4ec8a7c3be229df0055c (127.1.155.131:33277), b67e25d810534513892c093fa33600a6 (127.1.155.129:42235)
I20260620 21:45:29.683820 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760f3c6a6c044acbbb804960788982b1" candidate_uuid: "b4e8be01536a49b1a033922d9c8c7e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35c8149faea4ec8a7c3be229df0055c"
I20260620 21:45:29.683910 3530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760f3c6a6c044acbbb804960788982b1" candidate_uuid: "b4e8be01536a49b1a033922d9c8c7e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67e25d810534513892c093fa33600a6"
I20260620 21:45:29.683959 3810 raft_consensus.cc:3037] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:29.684074 3530 raft_consensus.cc:3037] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:29.685024 3810 raft_consensus.cc:2454] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4e8be01536a49b1a033922d9c8c7e69 in term 1.
I20260620 21:45:29.685113 3530 raft_consensus.cc:2454] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4e8be01536a49b1a033922d9c8c7e69 in term 1.
I20260620 21:45:29.685340 3603 leader_election.cc:302] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e8be01536a49b1a033922d9c8c7e69, e35c8149faea4ec8a7c3be229df0055c; no voters:
I20260620 21:45:29.685526 3886 raft_consensus.cc:2781] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:29.685873 3886 raft_consensus.cc:686] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [term 1 LEADER]: Becoming Leader. State: Replica: b4e8be01536a49b1a033922d9c8c7e69, State: Running, Role: LEADER
I20260620 21:45:29.686195 3886 consensus_queue.cc:237] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } }
I20260620 21:45:29.689329 3372 catalog_manager.cc:5495] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4e8be01536a49b1a033922d9c8c7e69 (127.1.155.130). New cstate: current_term: 1 leader_uuid: "b4e8be01536a49b1a033922d9c8c7e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } health_report { overall_health: UNKNOWN } } }
Jun 20 21:45:29 dist-test-slave-5bvn krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991928, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:29.740283 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.731732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33258 (local address 127.1.155.190:38851)
0620 21:45:29.731932 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:29.731940 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:29.732130 (+ 190us) server_negotiation.cc:407] Connection header received
0620 21:45:29.732291 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:29.732295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:29.732341 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:29.732428 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:29.733054 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.733572 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.734441 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.734648 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.734759 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.736914 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:29.736918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:29.736922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:29.736947 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:29.738506 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.738999 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.739004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.739006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.739076 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:29.739351 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:29.739354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:29.739356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:29.739525 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:29.739632 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:29.739946 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 21:45:29.740151 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":62}
I20260620 21:45:29.740293 3896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.731638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:38851 (local address 127.0.0.1:33258)
0620 21:45:29.732040 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:29.732052 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:29.732191 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:29.732447 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:29.732450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:29.732854 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:29.732938 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.732944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.733685 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.733689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.734329 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.734338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:29.734760 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:29.734764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:29.734839 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:29.735426 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:29.735438 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:29.736783 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:29.738616 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:29.738628 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:29.738636 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:29.738891 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:29.739170 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:29.739173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:29.739175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:29.739254 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:29.739630 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:29.739637 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:29.739778 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 21:45:29.739904 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 21:45:29.740083 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":112,"threads_started":1}
I20260620 21:45:29.744998 3372 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:33258:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 21:45:29.745620 3372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260620 21:45:29.750278 3857 tablet.cc:2369] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:45:29.750417 3510 tablet_service.cc:1467] Processing CreateTablet for tablet 5d1bf7888c2444c4bcfb736aafeba2ae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd2ecd5a5e8a4e12a738e3d00beeabd5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:45:29.750449 3641 tablet_service.cc:1467] Processing CreateTablet for tablet 5d1bf7888c2444c4bcfb736aafeba2ae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd2ecd5a5e8a4e12a738e3d00beeabd5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:45:29.751029 3510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d1bf7888c2444c4bcfb736aafeba2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:29.751128 3641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d1bf7888c2444c4bcfb736aafeba2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:29.753621 3881 tablet_bootstrap.cc:492] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69: Bootstrap starting.
I20260620 21:45:29.753716 3880 tablet_bootstrap.cc:492] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6: Bootstrap starting.
I20260620 21:45:29.753407 3790 tablet_service.cc:1467] Processing CreateTablet for tablet 5d1bf7888c2444c4bcfb736aafeba2ae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd2ecd5a5e8a4e12a738e3d00beeabd5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:45:29.753993 3790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d1bf7888c2444c4bcfb736aafeba2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:29.754478 3881 tablet_bootstrap.cc:654] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:29.754637 3880 tablet_bootstrap.cc:654] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:29.755796 3881 tablet_bootstrap.cc:492] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69: No bootstrap required, opened a new log
I20260620 21:45:29.755887 3881 ts_tablet_manager.cc:1397] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:45:29.756011 3880 tablet_bootstrap.cc:492] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6: No bootstrap required, opened a new log
I20260620 21:45:29.756076 3880 ts_tablet_manager.cc:1397] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:45:29.756347 3881 raft_consensus.cc:348] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:29.756458 3881 raft_consensus.cc:374] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:29.756491 3881 raft_consensus.cc:729] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e8be01536a49b1a033922d9c8c7e69, State: Initialized, Role: FOLLOWER
I20260620 21:45:29.756533 3880 raft_consensus.cc:348] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:29.756628 3880 raft_consensus.cc:374] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:29.756635 3879 tablet_bootstrap.cc:492] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c: Bootstrap starting.
I20260620 21:45:29.756659 3880 raft_consensus.cc:729] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67e25d810534513892c093fa33600a6, State: Initialized, Role: FOLLOWER
I20260620 21:45:29.756611 3881 consensus_queue.cc:260] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:29.756772 3880 consensus_queue.cc:260] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:29.756893 3881 ts_tablet_manager.cc:1428] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:29.757021 3880 ts_tablet_manager.cc:1428] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:45:29.757603 3879 tablet_bootstrap.cc:654] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:29.757028 3887 tablet_replica.cc:439] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } }
I20260620 21:45:29.757674 3887 tablet_replica.cc:442] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:29.757930 3886 tablet_replica.cc:439] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } }
I20260620 21:45:29.758064 3886 tablet_replica.cc:442] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:29.758877 3879 tablet_bootstrap.cc:492] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c: No bootstrap required, opened a new log
I20260620 21:45:29.758980 3879 ts_tablet_manager.cc:1397] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 21:45:29.759416 3879 raft_consensus.cc:348] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:29.759550 3879 raft_consensus.cc:374] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:29.759585 3879 raft_consensus.cc:729] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35c8149faea4ec8a7c3be229df0055c, State: Initialized, Role: FOLLOWER
I20260620 21:45:29.759716 3879 consensus_queue.cc:260] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:29.760007 3885 tablet_replica.cc:439] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } }
I20260620 21:45:29.760138 3885 tablet_replica.cc:442] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:29.760432 3879 ts_tablet_manager.cc:1428] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260620 21:45:29.782582 3717 tablet.cc:2369] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:45:29.806205 3577 tablet.cc:2369] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:45:29.952998 3886 raft_consensus.cc:482] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:29.953136 3886 raft_consensus.cc:504] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:29.953540 3886 leader_election.cc:288] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b67e25d810534513892c093fa33600a6 (127.1.155.129:42235), e35c8149faea4ec8a7c3be229df0055c (127.1.155.131:33277)
I20260620 21:45:29.954005 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d1bf7888c2444c4bcfb736aafeba2ae" candidate_uuid: "b4e8be01536a49b1a033922d9c8c7e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35c8149faea4ec8a7c3be229df0055c" is_pre_election: true
I20260620 21:45:29.954005 3530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d1bf7888c2444c4bcfb736aafeba2ae" candidate_uuid: "b4e8be01536a49b1a033922d9c8c7e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67e25d810534513892c093fa33600a6" is_pre_election: true
I20260620 21:45:29.954154 3810 raft_consensus.cc:2454] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4e8be01536a49b1a033922d9c8c7e69 in term 0.
I20260620 21:45:29.954154 3530 raft_consensus.cc:2454] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4e8be01536a49b1a033922d9c8c7e69 in term 0.
I20260620 21:45:29.954474 3602 leader_election.cc:302] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e8be01536a49b1a033922d9c8c7e69, b67e25d810534513892c093fa33600a6; no voters:
I20260620 21:45:29.954644 3886 raft_consensus.cc:2781] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:29.954748 3886 raft_consensus.cc:482] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:29.954803 3886 raft_consensus.cc:3037] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:29.956067 3886 raft_consensus.cc:504] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:29.956472 3886 leader_election.cc:288] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [CANDIDATE]: Term 1 election: Requested vote from peers b67e25d810534513892c093fa33600a6 (127.1.155.129:42235), e35c8149faea4ec8a7c3be229df0055c (127.1.155.131:33277)
I20260620 21:45:29.956836 3530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d1bf7888c2444c4bcfb736aafeba2ae" candidate_uuid: "b4e8be01536a49b1a033922d9c8c7e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67e25d810534513892c093fa33600a6"
I20260620 21:45:29.956897 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d1bf7888c2444c4bcfb736aafeba2ae" candidate_uuid: "b4e8be01536a49b1a033922d9c8c7e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35c8149faea4ec8a7c3be229df0055c"
I20260620 21:45:29.957002 3530 raft_consensus.cc:3037] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:29.957017 3810 raft_consensus.cc:3037] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:29.957973 3810 raft_consensus.cc:2454] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4e8be01536a49b1a033922d9c8c7e69 in term 1.
I20260620 21:45:29.958045 3530 raft_consensus.cc:2454] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4e8be01536a49b1a033922d9c8c7e69 in term 1.
I20260620 21:45:29.958278 3603 leader_election.cc:302] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e8be01536a49b1a033922d9c8c7e69, e35c8149faea4ec8a7c3be229df0055c; no voters:
I20260620 21:45:29.958456 3886 raft_consensus.cc:2781] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:29.958559 3886 raft_consensus.cc:686] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [term 1 LEADER]: Becoming Leader. State: Replica: b4e8be01536a49b1a033922d9c8c7e69, State: Running, Role: LEADER
I20260620 21:45:29.958707 3886 consensus_queue.cc:237] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:29.959395 3889 tablet_replica.cc:439] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69: TxnStatusTablet state changed. Reason: New leader b4e8be01536a49b1a033922d9c8c7e69. Latest consensus state: current_term: 1 leader_uuid: "b4e8be01536a49b1a033922d9c8c7e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } }
I20260620 21:45:29.959512 3889 tablet_replica.cc:442] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:45:29.959730 3901 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 21:45:29.959983 3901 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 21:45:29.960532 3371 catalog_manager.cc:5495] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4e8be01536a49b1a033922d9c8c7e69 (127.1.155.130). New cstate: current_term: 1 leader_uuid: "b4e8be01536a49b1a033922d9c8c7e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } health_report { overall_health: UNKNOWN } } }
I20260620 21:45:30.002987 3896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.998613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:43753 (local address 127.0.0.1:48836)
0620 21:45:29.998853 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:29.998869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:29.998991 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:29.999362 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:29.999365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:29.999372 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:45:29.999466 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:29.999471 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.000535 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.000538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:30.001216 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:30.001225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.001716 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.001720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:30.001817 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:30.001937 (+ 120us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:45:30.002644 (+ 707us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:45:30.002660 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 21:45:30.002741 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 21:45:30.002864 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260620 21:45:30.003034 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:29.998876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48836 (local address 127.1.155.130:43753)
0620 21:45:29.999027 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:29.999032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:29.999064 (+ 32us) server_negotiation.cc:407] Connection header received
0620 21:45:29.999143 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:29.999146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:29.999210 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:29.999320 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:29.999590 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.000409 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.001358 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.001596 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.001703 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:30.002094 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:30.002534 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:30.002660 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:30.002815 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 21:45:30.002882 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:45:30.009178 3530 raft_consensus.cc:1261] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Refusing update from remote peer b4e8be01536a49b1a033922d9c8c7e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:30.009204 3810 raft_consensus.cc:1261] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Refusing update from remote peer b4e8be01536a49b1a033922d9c8c7e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:30.009968 3889 consensus_queue.cc:1035] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:30.010210 3886 consensus_queue.cc:1035] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:30.013902 3887 tablet_replica.cc:439] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6: TxnStatusTablet state changed. Reason: New leader b4e8be01536a49b1a033922d9c8c7e69. Latest consensus state: current_term: 1 leader_uuid: "b4e8be01536a49b1a033922d9c8c7e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } }
I20260620 21:45:30.014048 3887 tablet_replica.cc:442] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:30.014272 3885 tablet_replica.cc:439] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c: TxnStatusTablet state changed. Reason: New leader b4e8be01536a49b1a033922d9c8c7e69. Latest consensus state: current_term: 1 leader_uuid: "b4e8be01536a49b1a033922d9c8c7e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } }
I20260620 21:45:30.014381 3885 tablet_replica.cc:442] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:30.017767 3904 mvcc.cc:205] Tried to move back new op lower bound from 7299038945309966336 to 7299038945112465408. Current Snapshot: MvccSnapshot[applied={T|T < 7299038945309966336}]
I20260620 21:45:30.017939 3886 tablet_replica.cc:439] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4e8be01536a49b1a033922d9c8c7e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } }
I20260620 21:45:30.018057 3886 tablet_replica.cc:442] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:45:30.019008 3887 tablet_replica.cc:439] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4e8be01536a49b1a033922d9c8c7e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } }
I20260620 21:45:30.019119 3887 tablet_replica.cc:442] T 5d1bf7888c2444c4bcfb736aafeba2ae P b67e25d810534513892c093fa33600a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:30.019747 3906 mvcc.cc:205] Tried to move back new op lower bound from 7299038945309966336 to 7299038945112465408. Current Snapshot: MvccSnapshot[applied={T|T < 7299038945309966336}]
I20260620 21:45:30.019781 3886 tablet_replica.cc:439] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4e8be01536a49b1a033922d9c8c7e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } }
I20260620 21:45:30.019898 3886 tablet_replica.cc:442] T 5d1bf7888c2444c4bcfb736aafeba2ae P b4e8be01536a49b1a033922d9c8c7e69: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:45:30.021893 3885 tablet_replica.cc:439] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4e8be01536a49b1a033922d9c8c7e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } } }
I20260620 21:45:30.022012 3885 tablet_replica.cc:442] T 5d1bf7888c2444c4bcfb736aafeba2ae P e35c8149faea4ec8a7c3be229df0055c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:30.025336 3905 mvcc.cc:205] Tried to move back new op lower bound from 7299038945309966336 to 7299038945112465408. Current Snapshot: MvccSnapshot[applied={T|T < 7299038945309966336 or (T in {7299038945309966336})}]
I20260620 21:45:30.030951 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:30.027094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48838 (local address 127.1.155.130:43753)
0620 21:45:30.027203 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:30.027208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:30.027222 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:45:30.027337 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:30.027340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:30.027389 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:30.027475 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:30.027980 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.028757 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.029686 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.030055 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.030161 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:30.030485 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:30.030570 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:30.030636 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:30.030740 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 21:45:30.030806 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:45:30.043525 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:30.039608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:35901 (local address 127.1.155.130:43753)
0620 21:45:30.039762 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:30.039768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:30.040017 (+ 249us) server_negotiation.cc:407] Connection header received
0620 21:45:30.040127 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:30.040129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:30.040178 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:30.040256 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:30.040518 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.041114 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.042392 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.042601 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.042692 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:30.043055 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:30.043121 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:30.043175 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:30.043289 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 21:45:30.043355 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:45:30.043534 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:30.039541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:43753 (local address 127.1.155.130:35901)
0620 21:45:30.039909 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:30.039925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:30.040037 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:30.040284 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:30.040287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:30.040307 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:45:30.040434 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:30.040440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.041230 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.041233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:30.042259 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:30.042270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.042741 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.042745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:30.042844 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:30.042899 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:45:30.043201 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:45:30.043207 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:45:30.043272 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 21:45:30.043365 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":157,"threads_started":1}
I20260620 21:45:30.050655 3530 raft_consensus.cc:1261] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Refusing update from remote peer b4e8be01536a49b1a033922d9c8c7e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:30.051021 3810 raft_consensus.cc:1261] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Refusing update from remote peer b4e8be01536a49b1a033922d9c8c7e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:30.051048 3886 consensus_queue.cc:1035] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:30.051345 3886 consensus_queue.cc:1035] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:30.105187 3373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:30.105370 3373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:30.108052 3373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac: Sending DeleteTablet for 3 replicas of tablet 760f3c6a6c044acbbb804960788982b1
I20260620 21:45:30.108937 3790 tablet_service.cc:1514] Processing DeleteTablet for tablet 760f3c6a6c044acbbb804960788982b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:30 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:34656
I20260620 21:45:30.109136 3510 tablet_service.cc:1514] Processing DeleteTablet for tablet 760f3c6a6c044acbbb804960788982b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:30 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:33890
I20260620 21:45:30.109153 3650 tablet_service.cc:1514] Processing DeleteTablet for tablet 760f3c6a6c044acbbb804960788982b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:30 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:48822
I20260620 21:45:30.109576 3920 tablet_replica.cc:330] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c: stopping tablet replica
I20260620 21:45:30.109628 3921 tablet_replica.cc:330] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6: stopping tablet replica
I20260620 21:45:30.110014 3920 raft_consensus.cc:2229] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:30.110011 3921 raft_consensus.cc:2229] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:30.110282 3921 raft_consensus.cc:2258] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:30.110313 3920 raft_consensus.cc:2258] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:30.110584 3373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:45:30.111090 3373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:30.111963 3920 ts_tablet_manager.cc:1905] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:30.112010 3922 tablet_replica.cc:330] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69: stopping tablet replica
I20260620 21:45:30.112306 3922 raft_consensus.cc:2229] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:45:30.112696 3922 raft_consensus.cc:2258] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:30.113919 3922 ts_tablet_manager.cc:1905] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:30.115725 3920 ts_tablet_manager.cc:1918] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:45:30.115893 3920 log.cc:1196] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/760f3c6a6c044acbbb804960788982b1
I20260620 21:45:30.116310 3920 ts_tablet_manager.cc:1939] T 760f3c6a6c044acbbb804960788982b1 P e35c8149faea4ec8a7c3be229df0055c: Deleting consensus metadata
I20260620 21:45:30.117159 3922 ts_tablet_manager.cc:1918] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:45:30.117268 3922 log.cc:1196] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/760f3c6a6c044acbbb804960788982b1
I20260620 21:45:30.117637 3922 ts_tablet_manager.cc:1939] T 760f3c6a6c044acbbb804960788982b1 P b4e8be01536a49b1a033922d9c8c7e69: Deleting consensus metadata
I20260620 21:45:30.117849 3359 catalog_manager.cc:4841] TS e35c8149faea4ec8a7c3be229df0055c (127.1.155.131:33277): tablet 760f3c6a6c044acbbb804960788982b1 (table test-table [id=17d71033a7b54c84be33cd9b15fe8d13]) successfully deleted
I20260620 21:45:30.118891 3360 catalog_manager.cc:4841] TS b4e8be01536a49b1a033922d9c8c7e69 (127.1.155.130:43753): tablet 760f3c6a6c044acbbb804960788982b1 (table test-table [id=17d71033a7b54c84be33cd9b15fe8d13]) successfully deleted
I20260620 21:45:30.118920 3650 tablet_service.cc:1467] Processing CreateTablet for tablet 4c968ac7e3874a9696f6b8eae39d5a22 (DEFAULT_TABLE table=test-table [id=1d80ba60fe234b73b8a5d6c6006a804d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:30.119155 3650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c968ac7e3874a9696f6b8eae39d5a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:30.118701 3510 tablet_service.cc:1467] Processing CreateTablet for tablet 4c968ac7e3874a9696f6b8eae39d5a22 (DEFAULT_TABLE table=test-table [id=1d80ba60fe234b73b8a5d6c6006a804d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:30.119338 3790 tablet_service.cc:1467] Processing CreateTablet for tablet 4c968ac7e3874a9696f6b8eae39d5a22 (DEFAULT_TABLE table=test-table [id=1d80ba60fe234b73b8a5d6c6006a804d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:30.119474 3510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c968ac7e3874a9696f6b8eae39d5a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:30.119567 3790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c968ac7e3874a9696f6b8eae39d5a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:30.121331 3881 tablet_bootstrap.cc:492] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69: Bootstrap starting.
I20260620 21:45:30.121987 3879 tablet_bootstrap.cc:492] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c: Bootstrap starting.
I20260620 21:45:30.122069 3880 tablet_bootstrap.cc:492] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6: Bootstrap starting.
I20260620 21:45:30.122277 3881 tablet_bootstrap.cc:654] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:30.123030 3880 tablet_bootstrap.cc:654] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:30.123030 3879 tablet_bootstrap.cc:654] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:30.123754 3881 tablet_bootstrap.cc:492] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69: No bootstrap required, opened a new log
I20260620 21:45:30.123831 3881 ts_tablet_manager.cc:1397] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:45:30.124267 3881 raft_consensus.cc:348] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:30.124374 3881 raft_consensus.cc:374] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:30.124402 3881 raft_consensus.cc:729] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e8be01536a49b1a033922d9c8c7e69, State: Initialized, Role: FOLLOWER
I20260620 21:45:30.124516 3881 consensus_queue.cc:260] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:30.124754 3881 ts_tablet_manager.cc:1428] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:30.124915 3879 tablet_bootstrap.cc:492] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c: No bootstrap required, opened a new log
I20260620 21:45:30.125010 3879 ts_tablet_manager.cc:1397] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 21:45:30.125401 3880 tablet_bootstrap.cc:492] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6: No bootstrap required, opened a new log
I20260620 21:45:30.125483 3880 ts_tablet_manager.cc:1397] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:45:30.125480 3879 raft_consensus.cc:348] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:30.125592 3879 raft_consensus.cc:374] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:30.125633 3879 raft_consensus.cc:729] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35c8149faea4ec8a7c3be229df0055c, State: Initialized, Role: FOLLOWER
I20260620 21:45:30.125766 3879 consensus_queue.cc:260] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:30.126066 3879 ts_tablet_manager.cc:1428] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:30.126016 3880 raft_consensus.cc:348] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:30.126132 3880 raft_consensus.cc:374] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:30.126176 3880 raft_consensus.cc:729] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67e25d810534513892c093fa33600a6, State: Initialized, Role: FOLLOWER
I20260620 21:45:30.126329 3880 consensus_queue.cc:260] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:30.126639 3880 ts_tablet_manager.cc:1428] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:30.111806 3921 ts_tablet_manager.cc:1905] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:30.131790 3921 ts_tablet_manager.cc:1918] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:45:30.131914 3921 log.cc:1196] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/wal/wals/760f3c6a6c044acbbb804960788982b1
I20260620 21:45:30.132225 3921 ts_tablet_manager.cc:1939] T 760f3c6a6c044acbbb804960788982b1 P b67e25d810534513892c093fa33600a6: Deleting consensus metadata
I20260620 21:45:30.133397 3358 catalog_manager.cc:4841] TS b67e25d810534513892c093fa33600a6 (127.1.155.129:42235): tablet 760f3c6a6c044acbbb804960788982b1 (table test-table [id=17d71033a7b54c84be33cd9b15fe8d13]) successfully deleted
I20260620 21:45:30.243884 3887 raft_consensus.cc:482] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:30.244027 3887 raft_consensus.cc:504] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:30.245182 3887 leader_election.cc:288] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4e8be01536a49b1a033922d9c8c7e69 (127.1.155.130:43753), e35c8149faea4ec8a7c3be229df0055c (127.1.155.131:33277)
I20260620 21:45:30.249684 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:30.245390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:33277 (local address 127.1.155.129:55611)
0620 21:45:30.245719 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:30.245735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:30.245861 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:30.246206 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:30.246209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:30.246228 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:30.246336 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:30.246342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.247303 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.247308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:30.248350 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:30.248360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.249243 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.249247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:30.249335 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:30.249363 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:45:30.249435 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 21:45:30.249504 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":78,"threads_started":1}
I20260620 21:45:30.249980 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:30.245792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:52095 (local address 127.1.155.130:43753)
0620 21:45:30.245937 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:30.245941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:30.246524 (+ 583us) server_negotiation.cc:407] Connection header received
0620 21:45:30.246587 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:30.246590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:30.246640 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:30.246738 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:30.246990 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.247698 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.248881 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.249346 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.249462 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:30.249505 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:30.249756 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 21:45:30.249861 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 21:45:30.250079 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:30.245537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:55611 (local address 127.1.155.131:33277)
0620 21:45:30.245990 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:30.245994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:30.246006 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:45:30.246053 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:30.246055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:30.246091 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:30.246185 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:30.246483 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.247201 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.248491 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.249114 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.249249 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:30.249305 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:30.249571 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 21:45:30.249661 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":84,"threads_started":1}
I20260620 21:45:30.250396 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:30.245397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:43753 (local address 127.1.155.129:52095)
0620 21:45:30.246414 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:30.246429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:30.246514 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:30.246751 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:30.246753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:30.246771 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:30.246869 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:30.246875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.247831 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.247835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:30.248772 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:30.248779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.249458 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.249462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:30.249565 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:30.249590 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:45:30.250170 (+ 580us) client_negotiation.cc:240] Negotiation successful
0620 21:45:30.250248 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":3,"thread_start_us":107,"threads_started":1}
I20260620 21:45:30.250424 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c968ac7e3874a9696f6b8eae39d5a22" candidate_uuid: "b67e25d810534513892c093fa33600a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35c8149faea4ec8a7c3be229df0055c" is_pre_election: true
I20260620 21:45:30.250600 3810 raft_consensus.cc:2454] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b67e25d810534513892c093fa33600a6 in term 0.
I20260620 21:45:30.250888 3670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c968ac7e3874a9696f6b8eae39d5a22" candidate_uuid: "b67e25d810534513892c093fa33600a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e8be01536a49b1a033922d9c8c7e69" is_pre_election: true
I20260620 21:45:30.250952 3463 leader_election.cc:302] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b67e25d810534513892c093fa33600a6, e35c8149faea4ec8a7c3be229df0055c; no voters:
I20260620 21:45:30.251116 3670 raft_consensus.cc:2454] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b67e25d810534513892c093fa33600a6 in term 0.
I20260620 21:45:30.251221 3887 raft_consensus.cc:2781] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:30.251279 3887 raft_consensus.cc:482] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:30.251333 3887 raft_consensus.cc:3037] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:30.252450 3887 raft_consensus.cc:504] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:30.252785 3887 leader_election.cc:288] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [CANDIDATE]: Term 1 election: Requested vote from peers b4e8be01536a49b1a033922d9c8c7e69 (127.1.155.130:43753), e35c8149faea4ec8a7c3be229df0055c (127.1.155.131:33277)
I20260620 21:45:30.253046 3670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c968ac7e3874a9696f6b8eae39d5a22" candidate_uuid: "b67e25d810534513892c093fa33600a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e8be01536a49b1a033922d9c8c7e69"
I20260620 21:45:30.253213 3670 raft_consensus.cc:3037] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:30.254199 3670 raft_consensus.cc:2454] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b67e25d810534513892c093fa33600a6 in term 1.
I20260620 21:45:30.254441 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c968ac7e3874a9696f6b8eae39d5a22" candidate_uuid: "b67e25d810534513892c093fa33600a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35c8149faea4ec8a7c3be229df0055c"
I20260620 21:45:30.254508 3464 leader_election.cc:302] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e8be01536a49b1a033922d9c8c7e69, b67e25d810534513892c093fa33600a6; no voters:
I20260620 21:45:30.254581 3810 raft_consensus.cc:3037] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:30.254701 3887 raft_consensus.cc:2781] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:30.254920 3887 raft_consensus.cc:686] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [term 1 LEADER]: Becoming Leader. State: Replica: b67e25d810534513892c093fa33600a6, State: Running, Role: LEADER
I20260620 21:45:30.255167 3887 consensus_queue.cc:237] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:30.255683 3810 raft_consensus.cc:2454] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b67e25d810534513892c093fa33600a6 in term 1.
I20260620 21:45:30.257659 3371 catalog_manager.cc:5495] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b67e25d810534513892c093fa33600a6 (127.1.155.129). New cstate: current_term: 1 leader_uuid: "b67e25d810534513892c093fa33600a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } health_report { overall_health: UNKNOWN } } }
I20260620 21:45:30.288216 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:30.283466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.1.155.129:42235)
0620 21:45:30.283952 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:30.283959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:30.283973 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:45:30.284046 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:30.284048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:30.284109 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:30.284236 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:30.284476 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.285302 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.286378 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.286732 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.286831 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:30.287123 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:30.287623 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:30.287735 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:30.287843 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 21:45:30.287898 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":106,"threads_started":1}
I20260620 21:45:30.299350 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:30.295347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:43753 (local address 127.1.155.129:59755)
0620 21:45:30.295759 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:30.295773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:30.295878 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:30.296108 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:30.296110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:30.296118 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:45:30.296244 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:30.296250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.296972 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.296975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:30.298038 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:30.298047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.298608 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.298611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:30.298686 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:30.298734 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:45:30.299025 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:45:30.299033 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 21:45:30.299117 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 21:45:30.299198 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":107,"threads_started":1}
I20260620 21:45:30.299331 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:30.295467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:59755 (local address 127.1.155.130:43753)
0620 21:45:30.295589 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:30.295593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:30.295851 (+ 258us) server_negotiation.cc:407] Connection header received
0620 21:45:30.295969 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:30.295971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:30.296013 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:30.296074 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:30.296337 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.296876 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.298148 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.298358 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.298500 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:30.298880 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:30.298944 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:30.299022 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:30.299137 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 21:45:30.299212 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 21:45:30.305799 3810 raft_consensus.cc:1261] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Refusing update from remote peer b67e25d810534513892c093fa33600a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:30.306010 3670 raft_consensus.cc:1261] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Refusing update from remote peer b67e25d810534513892c093fa33600a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:30.306306 3929 consensus_queue.cc:1035] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:30.306524 3887 consensus_queue.cc:1035] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:30.322988 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:30.319859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:41975 (local address 127.1.155.129:42235)
0620 21:45:30.319968 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:30.319972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:30.319983 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:45:30.320150 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:30.320153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:30.320206 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:30.320280 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:30.320525 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.321033 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.321880 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.322189 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.322289 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:30.322522 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:30.322620 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:30.322701 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:30.322782 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 21:45:30.322840 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 21:45:30.323024 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:30.319796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:42235 (local address 127.1.155.130:41975)
0620 21:45:30.319903 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:30.319918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:30.319990 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:30.320310 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:30.320313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:30.320321 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:45:30.320422 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:30.320427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.321141 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.321146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:30.321742 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:30.321750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:30.322284 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:30.322288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:30.322380 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:30.322405 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:45:30.322702 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:45:30.322708 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:45:30.322756 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 21:45:30.322865 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 21:45:30.323853 3906 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 21:45:31.878602 3371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:31.878747 3371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:31.880987 3371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac: Sending DeleteTablet for 3 replicas of tablet 4c968ac7e3874a9696f6b8eae39d5a22
I20260620 21:45:31.881435 3510 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c968ac7e3874a9696f6b8eae39d5a22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:31 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:33890
I20260620 21:45:31.881435 3641 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c968ac7e3874a9696f6b8eae39d5a22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:31 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:48822
I20260620 21:45:31.881593 3790 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c968ac7e3874a9696f6b8eae39d5a22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:31 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:34656
I20260620 21:45:31.881978 3963 tablet_replica.cc:330] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6: stopping tablet replica
I20260620 21:45:31.882170 3963 raft_consensus.cc:2229] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:45:31.881768 3962 tablet_replica.cc:330] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69: stopping tablet replica
I20260620 21:45:31.882465 3962 raft_consensus.cc:2229] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:31.882503 3963 raft_consensus.cc:2258] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:31.882624 3962 raft_consensus.cc:2258] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:31.883198 3963 ts_tablet_manager.cc:1905] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:31.883230 3962 ts_tablet_manager.cc:1905] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:31.885875 3963 ts_tablet_manager.cc:1918] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:45:31.885890 3962 ts_tablet_manager.cc:1918] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:45:31.885955 3962 log.cc:1196] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/4c968ac7e3874a9696f6b8eae39d5a22
I20260620 21:45:31.885984 3964 tablet_replica.cc:330] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c: stopping tablet replica
I20260620 21:45:31.886016 3963 log.cc:1196] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/wal/wals/4c968ac7e3874a9696f6b8eae39d5a22
I20260620 21:45:31.886294 3962 ts_tablet_manager.cc:1939] T 4c968ac7e3874a9696f6b8eae39d5a22 P b4e8be01536a49b1a033922d9c8c7e69: Deleting consensus metadata
I20260620 21:45:31.886302 3963 ts_tablet_manager.cc:1939] T 4c968ac7e3874a9696f6b8eae39d5a22 P b67e25d810534513892c093fa33600a6: Deleting consensus metadata
I20260620 21:45:31.886550 3964 raft_consensus.cc:2229] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:31.886761 3964 raft_consensus.cc:2258] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:31.886955 3360 catalog_manager.cc:4841] TS b4e8be01536a49b1a033922d9c8c7e69 (127.1.155.130:43753): tablet 4c968ac7e3874a9696f6b8eae39d5a22 (table test-table [id=1d80ba60fe234b73b8a5d6c6006a804d]) successfully deleted
I20260620 21:45:31.886955 3358 catalog_manager.cc:4841] TS b67e25d810534513892c093fa33600a6 (127.1.155.129:42235): tablet 4c968ac7e3874a9696f6b8eae39d5a22 (table test-table [id=1d80ba60fe234b73b8a5d6c6006a804d]) successfully deleted
I20260620 21:45:31.887564 3964 ts_tablet_manager.cc:1905] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:31.890000 3964 ts_tablet_manager.cc:1918] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:45:31.890094 3964 log.cc:1196] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/4c968ac7e3874a9696f6b8eae39d5a22
I20260620 21:45:31.890401 3964 ts_tablet_manager.cc:1939] T 4c968ac7e3874a9696f6b8eae39d5a22 P e35c8149faea4ec8a7c3be229df0055c: Deleting consensus metadata
I20260620 21:45:31.891278 3359 catalog_manager.cc:4841] TS e35c8149faea4ec8a7c3be229df0055c (127.1.155.131:33277): tablet 4c968ac7e3874a9696f6b8eae39d5a22 (table test-table [id=1d80ba60fe234b73b8a5d6c6006a804d]) successfully deleted
Jun 20 21:45:31 dist-test-slave-5bvn krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:45:31 dist-test-slave-5bvn krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:31.911094 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:31.902389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.1.155.190:38851)
0620 21:45:31.902661 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:31.902668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:31.902748 (+ 80us) server_negotiation.cc:407] Connection header received
0620 21:45:31.902854 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:31.902856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:31.902898 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:31.902977 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:31.903710 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:31.904256 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:31.905297 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:31.905516 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:31.905594 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:31.907936 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:31.907940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:31.907945 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:31.907972 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:31.909291 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:31.909711 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:31.909716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:31.909718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:31.909800 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:31.910226 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:31.910230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:31.910232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:31.910412 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:31.910553 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:31.910791 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 21:45:31.910913 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260620 21:45:31.913736 3371 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:45:31.914220 3371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:31.918593 3510 tablet_service.cc:1467] Processing CreateTablet for tablet 738659cef5d64ad5a1305f3e9666da8d (DEFAULT_TABLE table=test-table [id=f527200c1ed94a5d9b9d760c4cc65d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:31.918593 3641 tablet_service.cc:1467] Processing CreateTablet for tablet 738659cef5d64ad5a1305f3e9666da8d (DEFAULT_TABLE table=test-table [id=f527200c1ed94a5d9b9d760c4cc65d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:31.918731 3790 tablet_service.cc:1467] Processing CreateTablet for tablet 738659cef5d64ad5a1305f3e9666da8d (DEFAULT_TABLE table=test-table [id=f527200c1ed94a5d9b9d760c4cc65d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:31.918919 3641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 738659cef5d64ad5a1305f3e9666da8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:31.918919 3510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 738659cef5d64ad5a1305f3e9666da8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:31.919026 3790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 738659cef5d64ad5a1305f3e9666da8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:31.921089 3973 tablet_bootstrap.cc:492] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c: Bootstrap starting.
I20260620 21:45:31.921748 3974 tablet_bootstrap.cc:492] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6: Bootstrap starting.
I20260620 21:45:31.922335 3973 tablet_bootstrap.cc:654] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:31.922590 3974 tablet_bootstrap.cc:654] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:31.923300 3975 tablet_bootstrap.cc:492] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69: Bootstrap starting.
I20260620 21:45:31.923673 3973 tablet_bootstrap.cc:492] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c: No bootstrap required, opened a new log
I20260620 21:45:31.923763 3973 ts_tablet_manager.cc:1397] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:45:31.924027 3974 tablet_bootstrap.cc:492] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6: No bootstrap required, opened a new log
I20260620 21:45:31.924106 3974 ts_tablet_manager.cc:1397] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:45:31.924238 3973 raft_consensus.cc:348] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:31.924376 3973 raft_consensus.cc:374] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:31.924412 3973 raft_consensus.cc:729] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35c8149faea4ec8a7c3be229df0055c, State: Initialized, Role: FOLLOWER
I20260620 21:45:31.924494 3974 raft_consensus.cc:348] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:31.924595 3974 raft_consensus.cc:374] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:31.924615 3975 tablet_bootstrap.cc:654] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:31.924633 3974 raft_consensus.cc:729] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67e25d810534513892c093fa33600a6, State: Initialized, Role: FOLLOWER
I20260620 21:45:31.924585 3973 consensus_queue.cc:260] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:31.924779 3974 consensus_queue.cc:260] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:31.924986 3973 ts_tablet_manager.cc:1428] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:31.925473 3974 ts_tablet_manager.cc:1428] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:31.926671 3975 tablet_bootstrap.cc:492] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69: No bootstrap required, opened a new log
I20260620 21:45:31.926776 3975 ts_tablet_manager.cc:1397] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 21:45:31.927282 3975 raft_consensus.cc:348] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:31.927419 3975 raft_consensus.cc:374] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:31.927457 3975 raft_consensus.cc:729] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e8be01536a49b1a033922d9c8c7e69, State: Initialized, Role: FOLLOWER
I20260620 21:45:31.927587 3975 consensus_queue.cc:260] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:31.927927 3975 ts_tablet_manager.cc:1428] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:45:32.105510 3978 raft_consensus.cc:482] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:32.105719 3978 raft_consensus.cc:504] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:32.106832 3978 leader_election.cc:288] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b67e25d810534513892c093fa33600a6 (127.1.155.129:42235), b4e8be01536a49b1a033922d9c8c7e69 (127.1.155.130:43753)
I20260620 21:45:32.111733 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:32.107333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:45937 (local address 127.1.155.130:43753)
0620 21:45:32.107598 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:32.107609 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:32.107860 (+ 251us) server_negotiation.cc:407] Connection header received
0620 21:45:32.108071 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:32.108075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:32.108124 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:32.108236 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:32.108672 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.109372 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.110612 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.111146 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.111290 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:32.111341 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:32.111501 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 21:45:32.111596 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":83,"threads_started":1}
I20260620 21:45:32.111829 3982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:32.107290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:47433 (local address 127.1.155.129:42235)
0620 21:45:32.107630 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:32.107639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:32.107690 (+ 51us) server_negotiation.cc:407] Connection header received
0620 21:45:32.107853 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:32.107856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:32.107911 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:32.108004 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:32.108384 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.109124 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.110487 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.111121 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.111252 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:32.111308 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:32.111608 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 21:45:32.111693 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":117,"threads_started":1}
I20260620 21:45:32.111742 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:32.107144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:43753 (local address 127.1.155.131:45937)
0620 21:45:32.107700 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:32.107716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:32.107863 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:32.108327 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:32.108333 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:32.108357 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:32.108478 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:32.108485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.109540 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.109550 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:32.110483 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:32.110493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.111271 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.111275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:32.111375 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:32.111398 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:45:32.111478 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 21:45:32.111567 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":123,"thread_start_us":73,"threads_started":1}
I20260620 21:45:32.111747 3981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:32.107144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:42235 (local address 127.1.155.131:47433)
0620 21:45:32.107595 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:32.107611 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:32.107732 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:32.108059 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:32.108062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:32.108082 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:32.108245 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:32.108253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.109268 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.109271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:32.110336 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:32.110345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.111357 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.111360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:32.111476 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:32.111495 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 21:45:32.111558 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 21:45:32.111606 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":142,"threads_started":1}
I20260620 21:45:32.112356 3670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "738659cef5d64ad5a1305f3e9666da8d" candidate_uuid: "e35c8149faea4ec8a7c3be229df0055c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e8be01536a49b1a033922d9c8c7e69" is_pre_election: true
I20260620 21:45:32.112545 3670 raft_consensus.cc:2454] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35c8149faea4ec8a7c3be229df0055c in term 0.
I20260620 21:45:32.112541 3530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "738659cef5d64ad5a1305f3e9666da8d" candidate_uuid: "e35c8149faea4ec8a7c3be229df0055c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67e25d810534513892c093fa33600a6" is_pre_election: true
I20260620 21:45:32.112666 3530 raft_consensus.cc:2454] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35c8149faea4ec8a7c3be229df0055c in term 0.
I20260620 21:45:32.112964 3744 leader_election.cc:302] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e8be01536a49b1a033922d9c8c7e69, e35c8149faea4ec8a7c3be229df0055c; no voters:
I20260620 21:45:32.113281 3978 raft_consensus.cc:2781] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:32.113340 3978 raft_consensus.cc:482] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:32.113379 3978 raft_consensus.cc:3037] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:32.114444 3978 raft_consensus.cc:504] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:32.114800 3978 leader_election.cc:288] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [CANDIDATE]: Term 1 election: Requested vote from peers b67e25d810534513892c093fa33600a6 (127.1.155.129:42235), b4e8be01536a49b1a033922d9c8c7e69 (127.1.155.130:43753)
I20260620 21:45:32.115147 3530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "738659cef5d64ad5a1305f3e9666da8d" candidate_uuid: "e35c8149faea4ec8a7c3be229df0055c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67e25d810534513892c093fa33600a6"
I20260620 21:45:32.115149 3670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "738659cef5d64ad5a1305f3e9666da8d" candidate_uuid: "e35c8149faea4ec8a7c3be229df0055c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e8be01536a49b1a033922d9c8c7e69"
I20260620 21:45:32.115288 3530 raft_consensus.cc:3037] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:32.115278 3670 raft_consensus.cc:3037] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:32.116222 3530 raft_consensus.cc:2454] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35c8149faea4ec8a7c3be229df0055c in term 1.
I20260620 21:45:32.116235 3670 raft_consensus.cc:2454] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35c8149faea4ec8a7c3be229df0055c in term 1.
I20260620 21:45:32.116528 3744 leader_election.cc:302] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e8be01536a49b1a033922d9c8c7e69, e35c8149faea4ec8a7c3be229df0055c; no voters:
I20260620 21:45:32.116708 3978 raft_consensus.cc:2781] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:32.116946 3978 raft_consensus.cc:686] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [term 1 LEADER]: Becoming Leader. State: Replica: e35c8149faea4ec8a7c3be229df0055c, State: Running, Role: LEADER
I20260620 21:45:32.117224 3978 consensus_queue.cc:237] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:32.119630 3373 catalog_manager.cc:5495] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c reported cstate change: term changed from 0 to 1, leader changed from <none> to e35c8149faea4ec8a7c3be229df0055c (127.1.155.131). New cstate: current_term: 1 leader_uuid: "e35c8149faea4ec8a7c3be229df0055c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } health_report { overall_health: HEALTHY } } }
I20260620 21:45:32.169986 3987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:32.166118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34666 (local address 127.1.155.131:33277)
0620 21:45:32.166378 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:32.166385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:32.166397 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:45:32.166466 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:32.166468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:32.166518 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:32.166630 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:32.166878 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.167415 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.168286 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.168605 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.168696 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:32.168994 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:32.169485 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:32.169596 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:32.169710 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 21:45:32.169760 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":123,"threads_started":1}
I20260620 21:45:32.179905 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:32.176686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:53279 (local address 127.1.155.130:43753)
0620 21:45:32.176817 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:32.176822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:32.176926 (+ 104us) server_negotiation.cc:407] Connection header received
0620 21:45:32.177083 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:32.177087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:32.177139 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:32.177222 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:32.177464 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.178001 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.178869 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.179082 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.179189 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:32.179475 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:32.179547 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:32.179635 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:32.179717 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 21:45:32.179766 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:45:32.179929 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:32.176454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:43753 (local address 127.1.155.131:53279)
0620 21:45:32.176841 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:32.176855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:32.176982 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:32.177232 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:32.177235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:32.177243 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:45:32.177362 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:32.177367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.178119 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.178122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:32.178741 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:32.178749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.179190 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.179194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:32.179301 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:32.179341 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:45:32.179636 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:45:32.179644 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 21:45:32.179709 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 21:45:32.179786 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
I20260620 21:45:32.185981 3530 raft_consensus.cc:1261] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Refusing update from remote peer e35c8149faea4ec8a7c3be229df0055c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:32.186048 3670 raft_consensus.cc:1261] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Refusing update from remote peer e35c8149faea4ec8a7c3be229df0055c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:32.186455 3984 consensus_queue.cc:1035] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:32.186656 3978 consensus_queue.cc:1035] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:32.203979 3987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:32.200279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:57027 (local address 127.1.155.131:33277)
0620 21:45:32.200376 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:32.200379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:32.200691 (+ 312us) server_negotiation.cc:407] Connection header received
0620 21:45:32.200749 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:32.200768 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:32.200811 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:32.200868 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:32.201100 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.201767 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.202994 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.203242 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.203326 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:32.203604 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:32.203668 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:32.203722 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:32.203838 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 21:45:32.203879 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 21:45:32.204082 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:32.200240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:33277 (local address 127.1.155.130:57027)
0620 21:45:32.200498 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:32.200516 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:32.200625 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:32.200914 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:32.200917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:32.200927 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:45:32.201027 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:32.201031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.201909 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.201916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:32.202863 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:32.202871 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.203354 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.203357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:32.203424 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:32.203492 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:45:32.203743 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:45:32.203748 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:45:32.203835 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 21:45:32.203924 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":103,"threads_started":1}
I20260620 21:45:32.226419 3373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:32.226614 3373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:32.228996 3373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac: Sending DeleteTablet for 3 replicas of tablet 738659cef5d64ad5a1305f3e9666da8d
I20260620 21:45:32.229637 3641 tablet_service.cc:1514] Processing DeleteTablet for tablet 738659cef5d64ad5a1305f3e9666da8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:32 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:48822
I20260620 21:45:32.229635 3510 tablet_service.cc:1514] Processing DeleteTablet for tablet 738659cef5d64ad5a1305f3e9666da8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:32 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:33890
I20260620 21:45:32.229897 3962 tablet_replica.cc:330] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69: stopping tablet replica
I20260620 21:45:32.230163 3963 tablet_replica.cc:330] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6: stopping tablet replica
I20260620 21:45:32.230202 3790 tablet_service.cc:1514] Processing DeleteTablet for tablet 738659cef5d64ad5a1305f3e9666da8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:32 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:34656
I20260620 21:45:32.230257 3962 raft_consensus.cc:2229] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:32.230301 3963 raft_consensus.cc:2229] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:32.230387 3964 tablet_replica.cc:330] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c: stopping tablet replica
I20260620 21:45:32.230418 3963 raft_consensus.cc:2258] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:32.230391 3962 raft_consensus.cc:2258] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:32.230486 3964 raft_consensus.cc:2229] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:45:32.230790 3964 raft_consensus.cc:2258] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:32.231351 3962 ts_tablet_manager.cc:1905] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:32.231540 3964 ts_tablet_manager.cc:1905] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:32.231742 3373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:45:32.232192 3373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:32.234210 3964 ts_tablet_manager.cc:1918] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:45:32.234299 3964 log.cc:1196] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/738659cef5d64ad5a1305f3e9666da8d
I20260620 21:45:32.234624 3962 ts_tablet_manager.cc:1918] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:45:32.234779 3962 log.cc:1196] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/738659cef5d64ad5a1305f3e9666da8d
I20260620 21:45:32.235055 3962 ts_tablet_manager.cc:1939] T 738659cef5d64ad5a1305f3e9666da8d P b4e8be01536a49b1a033922d9c8c7e69: Deleting consensus metadata
I20260620 21:45:32.235062 3964 ts_tablet_manager.cc:1939] T 738659cef5d64ad5a1305f3e9666da8d P e35c8149faea4ec8a7c3be229df0055c: Deleting consensus metadata
I20260620 21:45:32.235821 3360 catalog_manager.cc:4841] TS b4e8be01536a49b1a033922d9c8c7e69 (127.1.155.130:43753): tablet 738659cef5d64ad5a1305f3e9666da8d (table test-table [id=f527200c1ed94a5d9b9d760c4cc65d01]) successfully deleted
I20260620 21:45:32.236071 3963 ts_tablet_manager.cc:1905] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:32.236124 3359 catalog_manager.cc:4841] TS e35c8149faea4ec8a7c3be229df0055c (127.1.155.131:33277): tablet 738659cef5d64ad5a1305f3e9666da8d (table test-table [id=f527200c1ed94a5d9b9d760c4cc65d01]) successfully deleted
I20260620 21:45:32.238765 3510 tablet_service.cc:1467] Processing CreateTablet for tablet 99e36a2161284c38b7415339a8c39cdf (DEFAULT_TABLE table=test-table [id=29bbbd5a0de74c9980b2ff1522d9c0b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:32.239050 3510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e36a2161284c38b7415339a8c39cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:32.239034 3641 tablet_service.cc:1467] Processing CreateTablet for tablet 99e36a2161284c38b7415339a8c39cdf (DEFAULT_TABLE table=test-table [id=29bbbd5a0de74c9980b2ff1522d9c0b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:32.239276 3641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e36a2161284c38b7415339a8c39cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:32.240835 3974 tablet_bootstrap.cc:492] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6: Bootstrap starting.
I20260620 21:45:32.241101 3963 ts_tablet_manager.cc:1918] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:45:32.241169 3963 log.cc:1196] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-0/wal/wals/738659cef5d64ad5a1305f3e9666da8d
I20260620 21:45:32.241443 3963 ts_tablet_manager.cc:1939] T 738659cef5d64ad5a1305f3e9666da8d P b67e25d810534513892c093fa33600a6: Deleting consensus metadata
I20260620 21:45:32.241456 3975 tablet_bootstrap.cc:492] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69: Bootstrap starting.
I20260620 21:45:32.241784 3974 tablet_bootstrap.cc:654] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:32.241911 3790 tablet_service.cc:1467] Processing CreateTablet for tablet 99e36a2161284c38b7415339a8c39cdf (DEFAULT_TABLE table=test-table [id=29bbbd5a0de74c9980b2ff1522d9c0b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:32.242123 3358 catalog_manager.cc:4841] TS b67e25d810534513892c093fa33600a6 (127.1.155.129:42235): tablet 738659cef5d64ad5a1305f3e9666da8d (table test-table [id=f527200c1ed94a5d9b9d760c4cc65d01]) successfully deleted
I20260620 21:45:32.242157 3790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e36a2161284c38b7415339a8c39cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:32.242295 3975 tablet_bootstrap.cc:654] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:32.243049 3974 tablet_bootstrap.cc:492] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6: No bootstrap required, opened a new log
I20260620 21:45:32.243151 3974 ts_tablet_manager.cc:1397] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:45:32.243327 3975 tablet_bootstrap.cc:492] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69: No bootstrap required, opened a new log
I20260620 21:45:32.243397 3975 ts_tablet_manager.cc:1397] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 21:45:32.243582 3974 raft_consensus.cc:348] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:32.243690 3974 raft_consensus.cc:374] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:32.243721 3974 raft_consensus.cc:729] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67e25d810534513892c093fa33600a6, State: Initialized, Role: FOLLOWER
I20260620 21:45:32.243772 3975 raft_consensus.cc:348] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:32.243863 3975 raft_consensus.cc:374] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:32.243844 3974 consensus_queue.cc:260] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:32.243894 3975 raft_consensus.cc:729] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e8be01536a49b1a033922d9c8c7e69, State: Initialized, Role: FOLLOWER
I20260620 21:45:32.244009 3975 consensus_queue.cc:260] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:32.244114 3974 ts_tablet_manager.cc:1428] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:32.244266 3975 ts_tablet_manager.cc:1428] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:45:32.244810 3973 tablet_bootstrap.cc:492] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c: Bootstrap starting.
I20260620 21:45:32.245712 3973 tablet_bootstrap.cc:654] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:32.247025 3973 tablet_bootstrap.cc:492] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c: No bootstrap required, opened a new log
I20260620 21:45:32.247107 3973 ts_tablet_manager.cc:1397] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 21:45:32.247525 3973 raft_consensus.cc:348] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:32.247668 3973 raft_consensus.cc:374] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:32.247700 3973 raft_consensus.cc:729] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35c8149faea4ec8a7c3be229df0055c, State: Initialized, Role: FOLLOWER
I20260620 21:45:32.247814 3973 consensus_queue.cc:260] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:32.248059 3973 ts_tablet_manager.cc:1428] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:45:32.437481 3887 raft_consensus.cc:482] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:32.437619 3887 raft_consensus.cc:504] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:32.438143 3887 leader_election.cc:288] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4e8be01536a49b1a033922d9c8c7e69 (127.1.155.130:43753), e35c8149faea4ec8a7c3be229df0055c (127.1.155.131:33277)
I20260620 21:45:32.438587 3670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e36a2161284c38b7415339a8c39cdf" candidate_uuid: "b67e25d810534513892c093fa33600a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e8be01536a49b1a033922d9c8c7e69" is_pre_election: true
I20260620 21:45:32.438719 3670 raft_consensus.cc:2454] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b67e25d810534513892c093fa33600a6 in term 0.
I20260620 21:45:32.438747 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e36a2161284c38b7415339a8c39cdf" candidate_uuid: "b67e25d810534513892c093fa33600a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35c8149faea4ec8a7c3be229df0055c" is_pre_election: true
I20260620 21:45:32.438953 3810 raft_consensus.cc:2454] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b67e25d810534513892c093fa33600a6 in term 0.
I20260620 21:45:32.439078 3464 leader_election.cc:302] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e8be01536a49b1a033922d9c8c7e69, b67e25d810534513892c093fa33600a6; no voters:
I20260620 21:45:32.439284 3887 raft_consensus.cc:2781] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:32.439358 3887 raft_consensus.cc:482] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:32.439417 3887 raft_consensus.cc:3037] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:32.440410 3887 raft_consensus.cc:504] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:32.440801 3887 leader_election.cc:288] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [CANDIDATE]: Term 1 election: Requested vote from peers b4e8be01536a49b1a033922d9c8c7e69 (127.1.155.130:43753), e35c8149faea4ec8a7c3be229df0055c (127.1.155.131:33277)
I20260620 21:45:32.441154 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e36a2161284c38b7415339a8c39cdf" candidate_uuid: "b67e25d810534513892c093fa33600a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35c8149faea4ec8a7c3be229df0055c"
I20260620 21:45:32.441191 3670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e36a2161284c38b7415339a8c39cdf" candidate_uuid: "b67e25d810534513892c093fa33600a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e8be01536a49b1a033922d9c8c7e69"
I20260620 21:45:32.441284 3810 raft_consensus.cc:3037] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:32.441319 3670 raft_consensus.cc:3037] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:32.442345 3670 raft_consensus.cc:2454] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b67e25d810534513892c093fa33600a6 in term 1.
I20260620 21:45:32.442376 3810 raft_consensus.cc:2454] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b67e25d810534513892c093fa33600a6 in term 1.
I20260620 21:45:32.442673 3464 leader_election.cc:302] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e8be01536a49b1a033922d9c8c7e69, b67e25d810534513892c093fa33600a6; no voters:
I20260620 21:45:32.442839 3887 raft_consensus.cc:2781] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:32.443034 3887 raft_consensus.cc:686] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [term 1 LEADER]: Becoming Leader. State: Replica: b67e25d810534513892c093fa33600a6, State: Running, Role: LEADER
I20260620 21:45:32.443214 3887 consensus_queue.cc:237] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } }
I20260620 21:45:32.445065 3372 catalog_manager.cc:5495] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b67e25d810534513892c093fa33600a6 (127.1.155.129). New cstate: current_term: 1 leader_uuid: "b67e25d810534513892c093fa33600a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67e25d810534513892c093fa33600a6" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 } health_report { overall_health: UNKNOWN } } }
I20260620 21:45:32.494519 3982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:32.491688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33916 (local address 127.1.155.129:42235)
0620 21:45:32.491815 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:32.491818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:32.491830 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:45:32.491890 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:32.491893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:32.491948 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:32.492014 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:32.492253 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.492741 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.493596 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:32.493814 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:32.493913 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:32.494175 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:32.494239 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:32.494297 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:32.494368 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 21:45:32.494411 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:45:32.500908 3670 raft_consensus.cc:1261] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Refusing update from remote peer b67e25d810534513892c093fa33600a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:32.501291 3810 raft_consensus.cc:1261] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Refusing update from remote peer b67e25d810534513892c093fa33600a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:32.501276 3887 consensus_queue.cc:1035] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4e8be01536a49b1a033922d9c8c7e69" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:32.501623 3887 consensus_queue.cc:1035] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35c8149faea4ec8a7c3be229df0055c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 33277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 21:45:32.514165 3945 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 21:45:34.067514 3372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:34.067688 3372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:34.070402 3372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3eb34716a2ea4937986b9eb80c8f86ac: Sending DeleteTablet for 3 replicas of tablet 99e36a2161284c38b7415339a8c39cdf
I20260620 21:45:34.070900 3510 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e36a2161284c38b7415339a8c39cdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:34 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:33890
I20260620 21:45:34.070991 3641 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e36a2161284c38b7415339a8c39cdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:34 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:48822
I20260620 21:45:34.071249 4027 tablet_replica.cc:330] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6: stopping tablet replica
I20260620 21:45:34.071399 4028 tablet_replica.cc:330] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69: stopping tablet replica
I20260620 21:45:34.071396 4027 raft_consensus.cc:2229] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:45:34.071467 3790 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e36a2161284c38b7415339a8c39cdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:34 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:34656
I20260620 21:45:34.071547 4028 raft_consensus.cc:2229] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:34.071612 4027 raft_consensus.cc:2258] T 99e36a2161284c38b7415339a8c39cdf P b67e25d810534513892c093fa33600a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:34.071655 4028 raft_consensus.cc:2258] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:34.071781 4029 tablet_replica.cc:330] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c: stopping tablet replica
I20260620 21:45:34.071935 4029 raft_consensus.cc:2229] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:34.072077 4029 raft_consensus.cc:2258] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:34.072286 4028 ts_tablet_manager.cc:1905] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:34.072335 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 3445
I20260620 21:45:34.073360 4029 ts_tablet_manager.cc:1905] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:34.075932 4029 ts_tablet_manager.cc:1918] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:45:34.076018 4029 log.cc:1196] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/99e36a2161284c38b7415339a8c39cdf
I20260620 21:45:34.076287 4029 ts_tablet_manager.cc:1939] T 99e36a2161284c38b7415339a8c39cdf P e35c8149faea4ec8a7c3be229df0055c: Deleting consensus metadata
I20260620 21:45:34.077009 4028 ts_tablet_manager.cc:1918] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:45:34.077090 4028 log.cc:1196] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/99e36a2161284c38b7415339a8c39cdf
I20260620 21:45:34.077360 4028 ts_tablet_manager.cc:1939] T 99e36a2161284c38b7415339a8c39cdf P b4e8be01536a49b1a033922d9c8c7e69: Deleting consensus metadata
I20260620 21:45:34.077399 3359 catalog_manager.cc:4841] TS e35c8149faea4ec8a7c3be229df0055c (127.1.155.131:33277): tablet 99e36a2161284c38b7415339a8c39cdf (table test-table [id=29bbbd5a0de74c9980b2ff1522d9c0b9]) successfully deleted
I20260620 21:45:34.078055 3360 catalog_manager.cc:4841] TS b4e8be01536a49b1a033922d9c8c7e69 (127.1.155.130:43753): tablet 99e36a2161284c38b7415339a8c39cdf (table test-table [id=29bbbd5a0de74c9980b2ff1522d9c0b9]) successfully deleted
W20260620 21:45:34.079412 3358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.155.129:42235 (error 108)
I20260620 21:45:34.079561 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 3585
I20260620 21:45:34.087327 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 3725
I20260620 21:45:34.089164 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:34.087397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:42235 (local address 127.0.0.1:33922)
0620 21:45:34.088952 (+ 1555us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:34.089057 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1477,"thread_start_us":96,"threads_started":1}
W20260620 21:45:34.089382 3358 catalog_manager.cc:4584] TS b67e25d810534513892c093fa33600a6 (127.1.155.129:42235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99e36a2161284c38b7415339a8c39cdf: Network error: Client connection negotiation failed: client connection to 127.1.155.129:42235: connect: Connection refused (error 111)
I20260620 21:45:34.094051 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 3341
2026-06-20T21:45:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6657 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:45:34 dist-test-slave-5bvn krb5kdc[4034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:45:34 dist-test-slave-5bvn krb5kdc[4034](info): set up 2 sockets
Jun 20 21:45:34 dist-test-slave-5bvn krb5kdc[4034](info): commencing operation
krb5kdc: starting...
W20260620 21:45:35.138501 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:45:35 dist-test-slave-5bvn krb5kdc[4034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:45:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:45:35.264243 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:35431
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:35431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:35.373828 4047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:35.374178 4047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:35.374271 4047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:35.378249 4047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:35.378358 4047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:35.378400 4047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:35.378475 4047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:35.378551 4047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:45:35.383870 4047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:35431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:35431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:35.384932 4047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:35.385962 4047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:35.394558 4052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:35.394618 4047 server_base.cc:1034] running on GCE node
W20260620 21:45:35.394565 4055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:35.394575 4053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:35.395393 4047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:35.396520 4047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:35.397727 4047 hybrid_clock.cc:648] HybridClock initialized: now 1781991935397704 us; error 37 us; skew 500 ppm
Jun 20 21:45:35 dist-test-slave-5bvn krb5kdc[4034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991935, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:35.400555 4047 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:45:35.401381 4047 webserver.cc:460] Webserver started at http://127.1.155.190:38277/ using document root <none> and password file <none>
I20260620 21:45:35.401930 4047 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:35.401994 4047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:35.402163 4047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:35.404057 4047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "e4eff63efe864177b949f96c1b0cb683"
format_stamp: "Formatted at 2026-06-20 21:45:35 on dist-test-slave-5bvn"
server_key: "89144e4d7051d47b60d2f6d614436e07"
server_key_iv: "fc67c47a5264064ee2708cea17eb9f91"
server_key_version: "encryptionkey@0"
I20260620 21:45:35.404662 4047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "e4eff63efe864177b949f96c1b0cb683"
format_stamp: "Formatted at 2026-06-20 21:45:35 on dist-test-slave-5bvn"
server_key: "89144e4d7051d47b60d2f6d614436e07"
server_key_iv: "fc67c47a5264064ee2708cea17eb9f91"
server_key_version: "encryptionkey@0"
I20260620 21:45:35.408282 4047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:45:35.410709 4062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:35.411916 4047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:45:35.412010 4047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "e4eff63efe864177b949f96c1b0cb683"
format_stamp: "Formatted at 2026-06-20 21:45:35 on dist-test-slave-5bvn"
server_key: "89144e4d7051d47b60d2f6d614436e07"
server_key_iv: "fc67c47a5264064ee2708cea17eb9f91"
server_key_version: "encryptionkey@0"
I20260620 21:45:35.412091 4047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:35.424733 4047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:35.427660 4047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:35.427811 4047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:35.434473 4047 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:35431
I20260620 21:45:35.434491 4114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:35431 every 8 connection(s)
I20260620 21:45:35.435384 4047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:35.437997 4115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:35.440030 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 4047
I20260620 21:45:35.440120 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:35.440398 1646 external_mini_cluster.cc:1428] Setting key a33e64675a7bfe514af8dcfc3e69442d
I20260620 21:45:35.444860 4115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683: Bootstrap starting.
I20260620 21:45:35.447425 4115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:35.448146 4115 log.cc:824] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:45:35 dist-test-slave-5bvn krb5kdc[4034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:35.450081 4115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683: No bootstrap required, opened a new log
I20260620 21:45:35.453660 4115 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4eff63efe864177b949f96c1b0cb683" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35431 } }
I20260620 21:45:35.453919 4115 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:35.453997 4115 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4eff63efe864177b949f96c1b0cb683, State: Initialized, Role: FOLLOWER
I20260620 21:45:35.454356 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:35.441730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.1.155.190:35431)
0620 21:45:35.442182 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:35.442202 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:35.442245 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:45:35.442985 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:35.443008 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:35.443418 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:35.443787 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:35.444414 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:35.445853 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:35.446670 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:35.447340 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:35.447503 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:35.449997 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:35.450002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:35.450014 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:35.450039 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:35.451652 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:35.452077 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:35.452088 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:35.452092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:35.452196 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:35.452470 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:35.452473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:35.452475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:35.452831 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:35.453046 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:35.453266 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 21:45:35.453537 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":147,"threads_started":1}
I20260620 21:45:35.454468 4115 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4eff63efe864177b949f96c1b0cb683" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35431 } }
I20260620 21:45:35.454636 4115 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:35.454698 4115 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:35.454792 4115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:35.455819 4115 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4eff63efe864177b949f96c1b0cb683" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35431 } }
I20260620 21:45:35.456156 4115 leader_election.cc:302] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4eff63efe864177b949f96c1b0cb683; no voters:
I20260620 21:45:35.456494 4115 leader_election.cc:288] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:35.456789 4120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:35.456960 4120 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [term 1 LEADER]: Becoming Leader. State: Replica: e4eff63efe864177b949f96c1b0cb683, State: Running, Role: LEADER
I20260620 21:45:35.457269 4120 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4eff63efe864177b949f96c1b0cb683" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35431 } }
I20260620 21:45:35.457716 4115 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:35.458894 4121 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4eff63efe864177b949f96c1b0cb683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4eff63efe864177b949f96c1b0cb683" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35431 } } }
I20260620 21:45:35.459040 4121 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:35.459409 4122 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4eff63efe864177b949f96c1b0cb683. Latest consensus state: current_term: 1 leader_uuid: "e4eff63efe864177b949f96c1b0cb683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4eff63efe864177b949f96c1b0cb683" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35431 } } }
I20260620 21:45:35.459503 4129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:35.459564 4122 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683 [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:35.462733 4129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:35.467993 4129 catalog_manager.cc:1347] Generated new cluster ID: 668ca870981d4e15bc21054f5712fab3
I20260620 21:45:35.468053 4129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:35.474897 4129 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:35.475745 4129 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:35.481892 4129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4eff63efe864177b949f96c1b0cb683: Generated new TSK 0
I20260620 21:45:35.482635 4129 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:45:35.518313 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:35431
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:35.648634 4141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:35.648916 4141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:35.648962 4141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:35.653033 4141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:35.653102 4141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:35.653162 4141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:45:35.658494 4141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:35431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:35.659592 4141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:35.660600 4141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:35.669883 4147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:35.669857 4146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:35.670012 4141 server_base.cc:1034] running on GCE node
W20260620 21:45:35.669884 4149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:35.670401 4141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:35.670971 4141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:35.672192 4141 hybrid_clock.cc:648] HybridClock initialized: now 1781991935672156 us; error 46 us; skew 500 ppm
Jun 20 21:45:35 dist-test-slave-5bvn krb5kdc[4034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991935, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:35.675011 4141 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:45:35.675776 4141 webserver.cc:460] Webserver started at http://127.1.155.129:39021/ using document root <none> and password file <none>
I20260620 21:45:35.676257 4141 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:35.676311 4141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:35.676465 4141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:35.678408 4141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "3f089beaef4f4547b7c6e8b6e0bb9c75"
format_stamp: "Formatted at 2026-06-20 21:45:35 on dist-test-slave-5bvn"
server_key: "1f247430d998f42f301ec3781d1da398"
server_key_iv: "b20dc84ebaa688f21812d33b0bad1a89"
server_key_version: "encryptionkey@0"
I20260620 21:45:35.678898 4141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "3f089beaef4f4547b7c6e8b6e0bb9c75"
format_stamp: "Formatted at 2026-06-20 21:45:35 on dist-test-slave-5bvn"
server_key: "1f247430d998f42f301ec3781d1da398"
server_key_iv: "b20dc84ebaa688f21812d33b0bad1a89"
server_key_version: "encryptionkey@0"
I20260620 21:45:35.682363 4141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:45:35.684625 4156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:35.685779 4141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:45:35.685930 4141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "3f089beaef4f4547b7c6e8b6e0bb9c75"
format_stamp: "Formatted at 2026-06-20 21:45:35 on dist-test-slave-5bvn"
server_key: "1f247430d998f42f301ec3781d1da398"
server_key_iv: "b20dc84ebaa688f21812d33b0bad1a89"
server_key_version: "encryptionkey@0"
I20260620 21:45:35.686038 4141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:35.700810 4141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:35.703641 4141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:35.703841 4141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:35.704416 4141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:35.705327 4141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:35.705394 4141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:35.705466 4141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:35.705509 4141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:35.714351 4141 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:34563
I20260620 21:45:35.714372 4269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:34563 every 8 connection(s)
I20260620 21:45:35.715314 4141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
Jun 20 21:45:35 dist-test-slave-5bvn krb5kdc[4034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991935, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:35.725579 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 4141
I20260620 21:45:35.725699 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:45:35.726027 1646 external_mini_cluster.cc:1428] Setting key 350e5e1af3b2de051a34e952373789b2
I20260620 21:45:35.729025 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:35.717136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:59699 (local address 127.1.155.190:35431)
0620 21:45:35.717284 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:35.717289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:35.718278 (+ 989us) server_negotiation.cc:407] Connection header received
0620 21:45:35.719546 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:35.719548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:35.719594 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:35.719678 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:35.720794 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:35.721312 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:35.722482 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:35.722714 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:35.722814 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:35.725211 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:35.725214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:35.725216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:35.725241 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:35.726889 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:35.727425 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:35.727428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:35.727430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:35.727516 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:35.727905 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:35.727908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:35.727909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:35.728066 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:35.728140 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:35.728801 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 21:45:35.728910 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:45:35.730046 4272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:35.717577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:35431 (local address 127.1.155.129:59699)
0620 21:45:35.718128 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:35.718170 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:35.719349 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:35.719848 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:35.719854 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:35.720203 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:35.720657 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:35.720669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:35.721418 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:35.721421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:35.722327 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:35.722337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:35.722858 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:35.722861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:35.722995 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:35.723609 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:35.723626 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:35.725032 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:35.727000 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:35.727005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:35.727016 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:35.727332 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:35.727616 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:35.727619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:35.727621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:35.727809 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:35.728173 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:35.728182 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:35.728515 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 21:45:35.728739 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 21:45:35.729097 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":116,"threads_started":1}
I20260620 21:45:35.731236 4270 heartbeater.cc:347] Connected to a master server at 127.1.155.190:35431
I20260620 21:45:35.731483 4270 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:35.732016 4270 heartbeater.cc:511] Master 127.1.155.190:35431 requested a full tablet report, sending...
I20260620 21:45:35.733721 4079 ts_manager.cc:194] Registered new tserver with Master: 3f089beaef4f4547b7c6e8b6e0bb9c75 (127.1.155.129:34563)
I20260620 21:45:35.735431 4079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:59699
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:45:35.753629 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:35431
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:35.867004 4275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:35.867251 4275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:35.867295 4275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:35.871186 4275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:35.871234 4275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:35.871291 4275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:45:35.876482 4275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:35431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:35.877403 4275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:35.878393 4275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:35.887563 4280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:35.887583 4283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:35.887637 4281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:35.887907 4275 server_base.cc:1034] running on GCE node
I20260620 21:45:35.888435 4275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:35.889057 4275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:35.890283 4275 hybrid_clock.cc:648] HybridClock initialized: now 1781991935890270 us; error 25 us; skew 500 ppm
Jun 20 21:45:35 dist-test-slave-5bvn krb5kdc[4034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991935, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:35.892938 4275 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:45:35.893759 4275 webserver.cc:460] Webserver started at http://127.1.155.130:37503/ using document root <none> and password file <none>
I20260620 21:45:35.894332 4275 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:35.894392 4275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:35.894544 4275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:35.896430 4275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "a3f22b29edf047a9bef2fa901d9bb38a"
format_stamp: "Formatted at 2026-06-20 21:45:35 on dist-test-slave-5bvn"
server_key: "ab571c488bc2b27926fc08d1cd91e74e"
server_key_iv: "46b9c59702a99326d5a4095a9a195bbe"
server_key_version: "encryptionkey@0"
I20260620 21:45:35.896920 4275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "a3f22b29edf047a9bef2fa901d9bb38a"
format_stamp: "Formatted at 2026-06-20 21:45:35 on dist-test-slave-5bvn"
server_key: "ab571c488bc2b27926fc08d1cd91e74e"
server_key_iv: "46b9c59702a99326d5a4095a9a195bbe"
server_key_version: "encryptionkey@0"
I20260620 21:45:35.900614 4275 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 21:45:35.903160 4290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:35.904462 4275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:45:35.904584 4275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "a3f22b29edf047a9bef2fa901d9bb38a"
format_stamp: "Formatted at 2026-06-20 21:45:35 on dist-test-slave-5bvn"
server_key: "ab571c488bc2b27926fc08d1cd91e74e"
server_key_iv: "46b9c59702a99326d5a4095a9a195bbe"
server_key_version: "encryptionkey@0"
I20260620 21:45:35.904695 4275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:35.921458 4275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:35.924626 4275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:35.924839 4275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:35.925453 4275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:35.926352 4275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:35.926405 4275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:35.926476 4275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:35.926519 4275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:35.934954 4275 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:40149
I20260620 21:45:35.934978 4403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:40149 every 8 connection(s)
I20260620 21:45:35.935875 4275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:45:35.940075 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 4275
I20260620 21:45:35.940194 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:45:35.940549 1646 external_mini_cluster.cc:1428] Setting key 817d3662a1e898530cd622fbe7bbcd64
Jun 20 21:45:35 dist-test-slave-5bvn krb5kdc[4034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991935, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
I20260620 21:45:35.950991 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:35.937759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:46961 (local address 127.1.155.190:35431)
0620 21:45:35.937936 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:35.937939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:35.938902 (+ 963us) server_negotiation.cc:407] Connection header received
0620 21:45:35.940491 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:35.940493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:35.940545 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:35.940665 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:35.942280 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:35.942855 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:35.944427 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:35.944664 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:35.944789 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:35.947367 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:35.947371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:35.947373 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:35.947397 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:35.948819 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:35.949362 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:35.949365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:35.949367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:35.949486 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:35.949862 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:35.949865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:35.949867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:35.950049 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:35.950155 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:35.950694 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 21:45:35.950815 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
I20260620 21:45:35.952000 4406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:35.938284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:35431 (local address 127.1.155.130:46961)
0620 21:45:35.938746 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:35.938788 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:35.940186 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:35.940863 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:35.940872 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:35.941339 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:35.942088 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:35.942107 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:35.942999 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:35.943005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:35.944286 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:35.944297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:35.944827 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:35.944831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:35.945000 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:35.945624 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:35.945641 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:35.947188 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:35.948966 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:35.948972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:35.948987 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:35.949253 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:35.949591 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:35.949594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:35.949595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:35.949738 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:35.950163 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:35.950179 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:35.950444 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 21:45:35.950627 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 21:45:35.950949 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260620 21:45:35.953253 4404 heartbeater.cc:347] Connected to a master server at 127.1.155.190:35431
I20260620 21:45:35.953522 4404 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:35.954053 4404 heartbeater.cc:511] Master 127.1.155.190:35431 requested a full tablet report, sending...
I20260620 21:45:35.955134 4079 ts_manager.cc:194] Registered new tserver with Master: a3f22b29edf047a9bef2fa901d9bb38a (127.1.155.130:40149)
I20260620 21:45:35.956012 4079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:46961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:45:35.968338 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:35431
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:36.080340 4409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:36.080600 4409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:36.080643 4409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:36.084859 4409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:36.084915 4409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:36.084973 4409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:45:36.090255 4409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:35431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:36.091228 4409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:36.092146 4409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:36.101799 4415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:36.101925 4417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:36.101908 4414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:36.102182 4409 server_base.cc:1034] running on GCE node
I20260620 21:45:36.102593 4409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:36.103188 4409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:36.104393 4409 hybrid_clock.cc:648] HybridClock initialized: now 1781991936104374 us; error 29 us; skew 500 ppm
Jun 20 21:45:36 dist-test-slave-5bvn krb5kdc[4034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991936, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:36.107352 4409 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:45:36.108280 4409 webserver.cc:460] Webserver started at http://127.1.155.131:40403/ using document root <none> and password file <none>
I20260620 21:45:36.108888 4409 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:36.108978 4409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:36.109211 4409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:36.111224 4409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "88e3f934375b425a9d87a2c19a11e5a8"
format_stamp: "Formatted at 2026-06-20 21:45:36 on dist-test-slave-5bvn"
server_key: "e1e1a89a1342156663b5ccb6ce7a8407"
server_key_iv: "ee7ad4cb172257b3be2a41d05f347a1d"
server_key_version: "encryptionkey@0"
I20260620 21:45:36.111776 4409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "88e3f934375b425a9d87a2c19a11e5a8"
format_stamp: "Formatted at 2026-06-20 21:45:36 on dist-test-slave-5bvn"
server_key: "e1e1a89a1342156663b5ccb6ce7a8407"
server_key_iv: "ee7ad4cb172257b3be2a41d05f347a1d"
server_key_version: "encryptionkey@0"
I20260620 21:45:36.115474 4409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:45:36.117935 4424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:36.119081 4409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:45:36.119222 4409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "88e3f934375b425a9d87a2c19a11e5a8"
format_stamp: "Formatted at 2026-06-20 21:45:36 on dist-test-slave-5bvn"
server_key: "e1e1a89a1342156663b5ccb6ce7a8407"
server_key_iv: "ee7ad4cb172257b3be2a41d05f347a1d"
server_key_version: "encryptionkey@0"
I20260620 21:45:36.119329 4409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:36.130254 4409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:36.133651 4409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:36.133920 4409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:36.134558 4409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:36.135382 4409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:36.135429 4409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:36.135499 4409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:36.135529 4409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:36.146008 4409 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:43371
I20260620 21:45:36.146028 4537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:43371 every 8 connection(s)
I20260620 21:45:36.146914 4409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:45:36.154659 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 4409
I20260620 21:45:36.154762 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:45:36.155056 1646 external_mini_cluster.cc:1428] Setting key cbcb82b039683f4c499fe69ce450ae2d
Jun 20 21:45:36 dist-test-slave-5bvn krb5kdc[4034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991936, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:36.161101 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:36.148780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:48905 (local address 127.1.155.190:35431)
0620 21:45:36.148945 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:36.148950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:36.149747 (+ 797us) server_negotiation.cc:407] Connection header received
0620 21:45:36.151091 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:36.151094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:36.151142 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:36.151223 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:36.152379 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:36.152912 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:36.154133 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:36.154443 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:36.154570 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:36.157525 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:36.157529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:36.157532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:36.157566 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:36.158979 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:36.159509 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:36.159515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:36.159520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:36.159609 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:36.159953 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:36.159958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:36.159960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:36.160174 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:36.160286 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:36.160801 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 21:45:36.160933 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:45:36.161985 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:36.149159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:35431 (local address 127.1.155.131:48905)
0620 21:45:36.149596 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:36.149639 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:36.150902 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:36.151377 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:36.151383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:36.151733 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:36.152225 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:36.152238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:36.153071 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:36.153074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:36.154017 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:36.154026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:36.154627 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:36.154631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:36.154908 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:36.155576 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:36.155593 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:36.157361 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:36.159096 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:36.159103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:36.159119 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:36.159400 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:36.159706 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:36.159709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:36.159712 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:36.159854 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:36.160281 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:36.160288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:36.160546 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 21:45:36.160746 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 21:45:36.161080 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":100,"threads_started":1}
I20260620 21:45:36.163117 4538 heartbeater.cc:347] Connected to a master server at 127.1.155.190:35431
I20260620 21:45:36.163390 4538 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:36.163893 4538 heartbeater.cc:511] Master 127.1.155.190:35431 requested a full tablet report, sending...
I20260620 21:45:36.164927 4079 ts_manager.cc:194] Registered new tserver with Master: 88e3f934375b425a9d87a2c19a11e5a8 (127.1.155.131:43371)
I20260620 21:45:36.165512 4079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:48905
I20260620 21:45:36.169067 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 21:45:36.181735 4118 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:45:36.180388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.1.155.190:35431)
0620 21:45:36.180568 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:36.180572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:36.180651 (+ 79us) server_negotiation.cc:407] Connection header received
0620 21:45:36.180770 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:36.180772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:36.180813 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:36.180884 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:36.181587 (+ 703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50438: BlockingRecv error: recv got EOF from 127.0.0.1:50438 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 21:45:36.182930 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 4141
I20260620 21:45:36.188128 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 4275
I20260620 21:45:36.193895 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 4409
I20260620 21:45:36.199136 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 4047
2026-06-20T21:45:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2103 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:45:36 dist-test-slave-5bvn krb5kdc[4550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:45:36 dist-test-slave-5bvn krb5kdc[4550](info): set up 2 sockets
Jun 20 21:45:36 dist-test-slave-5bvn krb5kdc[4550](info): commencing operation
krb5kdc: starting...
W20260620 21:45:37.241797 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:45:37 dist-test-slave-5bvn krb5kdc[4550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:45:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:45:37.360335 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:35851
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:35851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:37.467391 4563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:37.467705 4563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:37.467780 4563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:37.471789 4563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:37.471877 4563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:37.471915 4563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:37.471978 4563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:37.472016 4563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:45:37.477555 4563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:35851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:35851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:37.478610 4563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:37.479590 4563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:37.487668 4569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:37.487675 4571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:37.487720 4563 server_base.cc:1034] running on GCE node
W20260620 21:45:37.487675 4568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:37.488449 4563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:37.489470 4563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:37.490622 4563 hybrid_clock.cc:648] HybridClock initialized: now 1781991937490607 us; error 31 us; skew 500 ppm
Jun 20 21:45:37 dist-test-slave-5bvn krb5kdc[4550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991937, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:37.497212 4563 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:45:37.498168 4563 webserver.cc:460] Webserver started at http://127.1.155.190:43353/ using document root <none> and password file <none>
I20260620 21:45:37.498708 4563 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:37.498766 4563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:37.498980 4563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:37.500933 4563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "711c064a57a94e04b26bfa61d571817f"
format_stamp: "Formatted at 2026-06-20 21:45:37 on dist-test-slave-5bvn"
server_key: "79e94288400405ba1386256fa63a76c0"
server_key_iv: "49b42a20432ae3a1f1f38b41f17c5158"
server_key_version: "encryptionkey@0"
I20260620 21:45:37.501466 4563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "711c064a57a94e04b26bfa61d571817f"
format_stamp: "Formatted at 2026-06-20 21:45:37 on dist-test-slave-5bvn"
server_key: "79e94288400405ba1386256fa63a76c0"
server_key_iv: "49b42a20432ae3a1f1f38b41f17c5158"
server_key_version: "encryptionkey@0"
I20260620 21:45:37.505031 4563 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:45:37.507469 4578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:37.508947 4563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:45:37.509074 4563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "711c064a57a94e04b26bfa61d571817f"
format_stamp: "Formatted at 2026-06-20 21:45:37 on dist-test-slave-5bvn"
server_key: "79e94288400405ba1386256fa63a76c0"
server_key_iv: "49b42a20432ae3a1f1f38b41f17c5158"
server_key_version: "encryptionkey@0"
I20260620 21:45:37.509181 4563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:37.534942 4563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:37.538336 4563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:37.538553 4563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:37.545300 4563 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:35851
I20260620 21:45:37.545583 4630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:35851 every 8 connection(s)
I20260620 21:45:37.546343 4563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:37.549142 4631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:37.554267 4631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f: Bootstrap starting.
I20260620 21:45:37.556512 4631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:37.556629 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 4563
I20260620 21:45:37.556768 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:37.557024 1646 external_mini_cluster.cc:1428] Setting key 53c368a26a2e2f9039ac0f458c105cea
I20260620 21:45:37.557353 4631 log.cc:824] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:37.559393 4631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f: No bootstrap required, opened a new log
I20260620 21:45:37.563508 4631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711c064a57a94e04b26bfa61d571817f" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35851 } }
I20260620 21:45:37.563777 4631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:37.563843 4631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 711c064a57a94e04b26bfa61d571817f, State: Initialized, Role: FOLLOWER
I20260620 21:45:37.564325 4631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711c064a57a94e04b26bfa61d571817f" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35851 } }
I20260620 21:45:37.564508 4631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:37.564592 4631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:37.564702 4631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [term 0 FOLLOWER]: Advancing to term 1
Jun 20 21:45:37 dist-test-slave-5bvn krb5kdc[4550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:37.565749 4631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711c064a57a94e04b26bfa61d571817f" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35851 } }
I20260620 21:45:37.566128 4631 leader_election.cc:302] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 711c064a57a94e04b26bfa61d571817f; no voters:
I20260620 21:45:37.566504 4631 leader_election.cc:288] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:37.566601 4636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:37.567149 4636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [term 1 LEADER]: Becoming Leader. State: Replica: 711c064a57a94e04b26bfa61d571817f, State: Running, Role: LEADER
I20260620 21:45:37.567525 4636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711c064a57a94e04b26bfa61d571817f" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35851 } }
I20260620 21:45:37.567860 4631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:37.569052 4637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "711c064a57a94e04b26bfa61d571817f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711c064a57a94e04b26bfa61d571817f" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35851 } } }
I20260620 21:45:37.569164 4637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:37.569489 4638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 711c064a57a94e04b26bfa61d571817f. Latest consensus state: current_term: 1 leader_uuid: "711c064a57a94e04b26bfa61d571817f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711c064a57a94e04b26bfa61d571817f" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35851 } } }
I20260620 21:45:37.569566 4638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:37.572139 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:37.558110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58856 (local address 127.1.155.190:35851)
0620 21:45:37.558515 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:37.558530 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:37.558564 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:45:37.559175 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:37.559190 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:37.559557 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:37.559824 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:37.560692 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:37.561629 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:37.562951 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:37.563408 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:37.563535 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:37.566429 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:37.566438 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:37.566456 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:37.566487 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:37.568508 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:37.569077 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:37.569085 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:37.569093 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:37.569226 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:37.569578 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:37.569585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:37.569588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:37.570150 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:37.570894 (+ 744us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:37.571016 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 21:45:37.571230 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":122,"threads_started":1}
W20260620 21:45:37.572814 4651 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:45:37.572899 4651 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:45:37.572994 4645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:37.573858 4645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:37.575124 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:45:37.579779 4645 catalog_manager.cc:1347] Generated new cluster ID: 23a0e3cc13874168a4e8cbd0f9e289ae
I20260620 21:45:37.579838 4645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:37.592362 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:37.587592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58860 (local address 127.1.155.190:35851)
0620 21:45:37.587714 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:37.587718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:37.587962 (+ 244us) server_negotiation.cc:407] Connection header received
0620 21:45:37.588120 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:37.588123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:37.588175 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:37.588261 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:37.588761 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:37.589304 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:37.590233 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:37.590473 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:37.590587 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:37.591420 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:37.591424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:37.591427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:37.591915 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:37.591943 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:37.591959 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:37.592062 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:37.592156 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 21:45:37.592216 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:45:37.607048 4645 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:37.608016 4645 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:37.612861 4645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 711c064a57a94e04b26bfa61d571817f: Generated new TSK 0
I20260620 21:45:37.613526 4645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:45:37.621237 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 4563
2026-06-20T21:45:37Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1419 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:45:37 dist-test-slave-5bvn krb5kdc[4664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:45:37 dist-test-slave-5bvn krb5kdc[4664](info): set up 2 sockets
Jun 20 21:45:37 dist-test-slave-5bvn krb5kdc[4664](info): commencing operation
krb5kdc: starting...
W20260620 21:45:38.669788 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:45:38 dist-test-slave-5bvn krb5kdc[4664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:45:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:45:38.788861 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:42559
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:44623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:42559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:38.899258 4677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:38.899566 4677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:38.899641 4677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:38.903744 4677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:38.903832 4677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:38.903869 4677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:38.903929 4677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:38.903986 4677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:45:38.909989 4677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:44623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:42559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:42559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:38.911110 4677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:38.912164 4677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:38.920845 4685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:38.920848 4682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:38.920852 4683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:38.920885 4677 server_base.cc:1034] running on GCE node
I20260620 21:45:38.921677 4677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:38.922888 4677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:38.924063 4677 hybrid_clock.cc:648] HybridClock initialized: now 1781991938924038 us; error 33 us; skew 500 ppm
Jun 20 21:45:38 dist-test-slave-5bvn krb5kdc[4664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991938, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:38.926894 4677 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:45:38.927852 4677 webserver.cc:460] Webserver started at http://127.1.155.190:43281/ using document root <none> and password file <none>
I20260620 21:45:38.928445 4677 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:38.928566 4677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:38.928804 4677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:38.930888 4677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "c78ec0c0103e48df8d943e23288f6c97"
format_stamp: "Formatted at 2026-06-20 21:45:38 on dist-test-slave-5bvn"
server_key: "139ec8d00b71cf2483df6bf3c4aff0fb"
server_key_iv: "8fc7281e230302987083e9d656e37b33"
server_key_version: "encryptionkey@0"
I20260620 21:45:38.931483 4677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "c78ec0c0103e48df8d943e23288f6c97"
format_stamp: "Formatted at 2026-06-20 21:45:38 on dist-test-slave-5bvn"
server_key: "139ec8d00b71cf2483df6bf3c4aff0fb"
server_key_iv: "8fc7281e230302987083e9d656e37b33"
server_key_version: "encryptionkey@0"
I20260620 21:45:38.935108 4677 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:45:38.937316 4692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:38.938509 4677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:45:38.938648 4677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "c78ec0c0103e48df8d943e23288f6c97"
format_stamp: "Formatted at 2026-06-20 21:45:38 on dist-test-slave-5bvn"
server_key: "139ec8d00b71cf2483df6bf3c4aff0fb"
server_key_iv: "8fc7281e230302987083e9d656e37b33"
server_key_version: "encryptionkey@0"
I20260620 21:45:38.938755 4677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:38.952745 4677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:38.956053 4677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:38.956269 4677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:38.963357 4677 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:42559
I20260620 21:45:38.963634 4744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:42559 every 8 connection(s)
I20260620 21:45:38.964393 4677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:38.965010 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 4677
I20260620 21:45:38.965154 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:38.965435 1646 external_mini_cluster.cc:1428] Setting key 39b4e2fa215be50ea9f541d9ee85dad1
I20260620 21:45:38.967660 4745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:38.975054 4745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97: Bootstrap starting.
Jun 20 21:45:38 dist-test-slave-5bvn krb5kdc[4664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:38.977715 4745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:38.978752 4745 log.cc:824] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:38.981343 4745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97: No bootstrap required, opened a new log
I20260620 21:45:38.982573 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:38.966889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49042 (local address 127.1.155.190:42559)
0620 21:45:38.967310 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:38.967330 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:38.967375 (+ 45us) server_negotiation.cc:407] Connection header received
0620 21:45:38.968237 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:38.968272 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:38.968674 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:38.969039 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:38.969710 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:38.971277 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:38.972181 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:38.972871 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:38.973052 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:38.976283 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:38.976291 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:38.976309 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:38.976349 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:38.979053 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:38.979746 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:38.979753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:38.979759 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:38.979939 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:38.980268 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:38.980271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:38.980273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:38.980786 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:38.980994 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:38.981259 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 21:45:38.981566 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":108,"threads_started":1}
I20260620 21:45:38.985675 4745 raft_consensus.cc:348] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78ec0c0103e48df8d943e23288f6c97" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42559 } }
I20260620 21:45:38.985963 4745 raft_consensus.cc:374] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:38.986020 4745 raft_consensus.cc:729] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c78ec0c0103e48df8d943e23288f6c97, State: Initialized, Role: FOLLOWER
I20260620 21:45:38.986501 4745 consensus_queue.cc:260] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78ec0c0103e48df8d943e23288f6c97" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42559 } }
I20260620 21:45:38.986629 4745 raft_consensus.cc:388] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:38.986670 4745 raft_consensus.cc:482] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:38.986747 4745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:38.987824 4745 raft_consensus.cc:504] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78ec0c0103e48df8d943e23288f6c97" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42559 } }
I20260620 21:45:38.988181 4745 leader_election.cc:302] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c78ec0c0103e48df8d943e23288f6c97; no voters:
I20260620 21:45:38.988507 4745 leader_election.cc:288] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:38.988730 4750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:38.989070 4750 raft_consensus.cc:686] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [term 1 LEADER]: Becoming Leader. State: Replica: c78ec0c0103e48df8d943e23288f6c97, State: Running, Role: LEADER
I20260620 21:45:38.989454 4750 consensus_queue.cc:237] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78ec0c0103e48df8d943e23288f6c97" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42559 } }
I20260620 21:45:38.989916 4745 sys_catalog.cc:564] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:38.991353 4751 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c78ec0c0103e48df8d943e23288f6c97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78ec0c0103e48df8d943e23288f6c97" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42559 } } }
I20260620 21:45:38.991494 4751 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:38.991945 4752 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c78ec0c0103e48df8d943e23288f6c97. Latest consensus state: current_term: 1 leader_uuid: "c78ec0c0103e48df8d943e23288f6c97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78ec0c0103e48df8d943e23288f6c97" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42559 } } }
I20260620 21:45:38.992077 4752 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97 [sys.catalog]: This master's current role is: LEADER
W20260620 21:45:38.995301 4765 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:45:38.995389 4765 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:45:38.995512 4761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:38.996286 4761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:39.001773 4761 catalog_manager.cc:1347] Generated new cluster ID: 70050ecf469e4560bdf6e67eaf658f55
I20260620 21:45:39.001876 4761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:39.044101 4761 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:39.045073 4761 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:39.050513 4761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c78ec0c0103e48df8d943e23288f6c97: Generated new TSK 0
I20260620 21:45:39.051216 4761 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:45:39.088732 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42559
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:44623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:39.200783 4771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:39.201050 4771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:39.201094 4771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:39.205236 4771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:39.205289 4771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:39.205394 4771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:45:39.210557 4771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:44623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:39.211529 4771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:39.212515 4771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:39.221827 4777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:39.221830 4779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:39.221827 4776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:39.222158 4771 server_base.cc:1034] running on GCE node
I20260620 21:45:39.222585 4771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:39.223259 4771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:39.224471 4771 hybrid_clock.cc:648] HybridClock initialized: now 1781991939224445 us; error 41 us; skew 500 ppm
Jun 20 21:45:39 dist-test-slave-5bvn krb5kdc[4664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991939, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:39.227164 4771 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:45:39.227993 4771 webserver.cc:460] Webserver started at http://127.1.155.129:46789/ using document root <none> and password file <none>
I20260620 21:45:39.228557 4771 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:39.228647 4771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:39.228885 4771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:39.230935 4771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "ffb5b82543ad411baa594737d57b7f10"
format_stamp: "Formatted at 2026-06-20 21:45:39 on dist-test-slave-5bvn"
server_key: "6feddad3f6b97edf13f671e7f96dc241"
server_key_iv: "80e572f21e3aedce8716a9f5f287be0e"
server_key_version: "encryptionkey@0"
I20260620 21:45:39.231496 4771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "ffb5b82543ad411baa594737d57b7f10"
format_stamp: "Formatted at 2026-06-20 21:45:39 on dist-test-slave-5bvn"
server_key: "6feddad3f6b97edf13f671e7f96dc241"
server_key_iv: "80e572f21e3aedce8716a9f5f287be0e"
server_key_version: "encryptionkey@0"
I20260620 21:45:39.235145 4771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:45:39.237461 4786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:39.238714 4771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:45:39.238875 4771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "ffb5b82543ad411baa594737d57b7f10"
format_stamp: "Formatted at 2026-06-20 21:45:39 on dist-test-slave-5bvn"
server_key: "6feddad3f6b97edf13f671e7f96dc241"
server_key_iv: "80e572f21e3aedce8716a9f5f287be0e"
server_key_version: "encryptionkey@0"
I20260620 21:45:39.239005 4771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:39.254762 4771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:39.257651 4771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:39.257941 4771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:39.258548 4771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:39.259364 4771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:39.259438 4771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:39.259507 4771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:39.259559 4771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:39.268491 4771 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:42863
I20260620 21:45:39.268513 4899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:42863 every 8 connection(s)
I20260620 21:45:39.269387 4771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:45:39.275664 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 4771
I20260620 21:45:39.275802 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:45:39.276129 1646 external_mini_cluster.cc:1428] Setting key 45c7f0f9dc9354f539dc5bcdd347e86b
Jun 20 21:45:39 dist-test-slave-5bvn krb5kdc[4664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991939, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:39.284008 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:39.271327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:32965 (local address 127.1.155.190:42559)
0620 21:45:39.271470 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:39.271474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:39.272252 (+ 778us) server_negotiation.cc:407] Connection header received
0620 21:45:39.273673 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:39.273677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:39.273735 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:39.273861 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:39.274989 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:39.275515 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:39.276802 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:39.277049 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:39.277161 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:39.279852 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:39.279858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:39.279861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:39.279893 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:39.281783 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:39.282323 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:39.282327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:39.282329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:39.282441 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:39.282770 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:39.282776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:39.282781 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:39.283069 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:39.283191 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:39.283663 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 21:45:39.283802 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:45:39.284760 4902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:39.271681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42559 (local address 127.1.155.129:32965)
0620 21:45:39.272110 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:39.272151 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:39.273458 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:39.274029 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:39.274036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:39.274390 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:39.274839 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:39.274852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:39.275642 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:39.275645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:39.276673 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:39.276682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:39.277197 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:39.277201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:39.277376 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:39.278008 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:39.278024 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:39.279667 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:39.281953 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:39.281958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:39.281970 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:39.282217 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:39.282549 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:39.282551 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:39.282552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:39.282674 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:39.283182 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:39.283187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:39.283422 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 21:45:39.283575 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 21:45:39.283855 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
I20260620 21:45:39.286134 4900 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42559
I20260620 21:45:39.286382 4900 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:39.286844 4900 heartbeater.cc:511] Master 127.1.155.190:42559 requested a full tablet report, sending...
I20260620 21:45:39.288358 4708 ts_manager.cc:194] Registered new tserver with Master: ffb5b82543ad411baa594737d57b7f10 (127.1.155.129:42863)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
I20260620 21:45:39.289670 4708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:32965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:45:39.305723 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42559
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:44623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:39.416620 4905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:39.416885 4905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:39.416927 4905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:39.421234 4905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:39.421295 4905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:39.421382 4905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:45:39.426929 4905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:44623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:39.428054 4905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:39.429088 4905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:39.438835 4911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:39.438829 4913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:39.438856 4910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:39.439043 4905 server_base.cc:1034] running on GCE node
I20260620 21:45:39.439495 4905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:39.440203 4905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:39.441394 4905 hybrid_clock.cc:648] HybridClock initialized: now 1781991939441380 us; error 28 us; skew 500 ppm
Jun 20 21:45:39 dist-test-slave-5bvn krb5kdc[4664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991939, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:39.444406 4905 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:45:39.445313 4905 webserver.cc:460] Webserver started at http://127.1.155.130:36867/ using document root <none> and password file <none>
I20260620 21:45:39.445977 4905 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:39.446074 4905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:39.446297 4905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:39.448333 4905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "7950f0750a184390a0d68bcbf3628fc4"
format_stamp: "Formatted at 2026-06-20 21:45:39 on dist-test-slave-5bvn"
server_key: "1d088eadac610a9569cf2c894dcc403d"
server_key_iv: "ec7adfead57a7317336543acdd8a5116"
server_key_version: "encryptionkey@0"
I20260620 21:45:39.448937 4905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "7950f0750a184390a0d68bcbf3628fc4"
format_stamp: "Formatted at 2026-06-20 21:45:39 on dist-test-slave-5bvn"
server_key: "1d088eadac610a9569cf2c894dcc403d"
server_key_iv: "ec7adfead57a7317336543acdd8a5116"
server_key_version: "encryptionkey@0"
I20260620 21:45:39.452819 4905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 21:45:39.455324 4920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:39.456522 4905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:45:39.456671 4905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "7950f0750a184390a0d68bcbf3628fc4"
format_stamp: "Formatted at 2026-06-20 21:45:39 on dist-test-slave-5bvn"
server_key: "1d088eadac610a9569cf2c894dcc403d"
server_key_iv: "ec7adfead57a7317336543acdd8a5116"
server_key_version: "encryptionkey@0"
I20260620 21:45:39.456782 4905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:39.472855 4905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:39.476130 4905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:39.476329 4905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:39.476900 4905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:39.477690 4905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:39.477762 4905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:39.477854 4905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:39.477908 4905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:39.487111 4905 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:44251
I20260620 21:45:39.487129 5033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:44251 every 8 connection(s)
I20260620 21:45:39.487980 4905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:45:39.492293 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 4905
I20260620 21:45:39.492426 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:45:39.492771 1646 external_mini_cluster.cc:1428] Setting key 3722a487864b20bf43e506a367e66a17
Jun 20 21:45:39 dist-test-slave-5bvn krb5kdc[4664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991939, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
I20260620 21:45:39.503937 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:39.489800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:46575 (local address 127.1.155.190:42559)
0620 21:45:39.490030 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:39.490033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:39.490896 (+ 863us) server_negotiation.cc:407] Connection header received
0620 21:45:39.492634 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:39.492639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:39.492696 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:39.492799 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:39.494340 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:39.495185 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:39.496963 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:39.497185 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:39.497287 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:39.500285 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:39.500289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:39.500294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:39.500323 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:39.501757 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:39.502370 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:39.502379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:39.502380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:39.502451 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:39.502809 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:39.502814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:39.502820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:39.503008 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:39.503114 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:39.503615 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 21:45:39.503762 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
I20260620 21:45:39.504797 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:39.490291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42559 (local address 127.1.155.130:46575)
0620 21:45:39.490743 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:39.490785 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:39.492353 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:39.493071 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:39.493080 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:39.493528 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:39.494171 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:39.494197 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:39.495324 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:39.495327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:39.496798 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:39.496813 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:39.497353 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:39.497360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:39.497606 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:39.498370 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:39.498399 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:39.500100 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:39.501898 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:39.501905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:39.501921 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:39.502250 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:39.502550 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:39.502553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:39.502555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:39.502700 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:39.503113 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:39.503120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:39.503376 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 21:45:39.503562 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 21:45:39.503889 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260620 21:45:39.506089 5034 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42559
I20260620 21:45:39.506389 5034 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:39.506990 5034 heartbeater.cc:511] Master 127.1.155.190:42559 requested a full tablet report, sending...
I20260620 21:45:39.508136 4708 ts_manager.cc:194] Registered new tserver with Master: 7950f0750a184390a0d68bcbf3628fc4 (127.1.155.130:44251)
I20260620 21:45:39.508821 4708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:46575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:45:39.522161 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42559
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:44623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:39.630460 5039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:39.630754 5039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:39.630846 5039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:39.634882 5039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:39.634981 5039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:39.635070 5039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:45:39.640359 5039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:44623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:39.641435 5039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:39.642568 5039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:39.652251 5047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:39.652235 5045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:39.652246 5044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:39.652650 5039 server_base.cc:1034] running on GCE node
I20260620 21:45:39.653066 5039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:39.653735 5039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:39.654937 5039 hybrid_clock.cc:648] HybridClock initialized: now 1781991939654900 us; error 47 us; skew 500 ppm
Jun 20 21:45:39 dist-test-slave-5bvn krb5kdc[4664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991939, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:39.657764 5039 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:45:39.658674 5039 webserver.cc:460] Webserver started at http://127.1.155.131:44315/ using document root <none> and password file <none>
I20260620 21:45:39.659189 5039 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:39.659246 5039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:39.659415 5039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:39.661531 5039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "7dbb8e724c5d4197b7ee0f41f11a22a3"
format_stamp: "Formatted at 2026-06-20 21:45:39 on dist-test-slave-5bvn"
server_key: "3eac5835f8408b565232c0a6915f5ece"
server_key_iv: "a47840ecdab79e0cb0b72179bcaceab9"
server_key_version: "encryptionkey@0"
I20260620 21:45:39.662110 5039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "7dbb8e724c5d4197b7ee0f41f11a22a3"
format_stamp: "Formatted at 2026-06-20 21:45:39 on dist-test-slave-5bvn"
server_key: "3eac5835f8408b565232c0a6915f5ece"
server_key_iv: "a47840ecdab79e0cb0b72179bcaceab9"
server_key_version: "encryptionkey@0"
I20260620 21:45:39.667986 5039 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260620 21:45:39.671734 5054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:39.674263 5039 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:45:39.674472 5039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "7dbb8e724c5d4197b7ee0f41f11a22a3"
format_stamp: "Formatted at 2026-06-20 21:45:39 on dist-test-slave-5bvn"
server_key: "3eac5835f8408b565232c0a6915f5ece"
server_key_iv: "a47840ecdab79e0cb0b72179bcaceab9"
server_key_version: "encryptionkey@0"
I20260620 21:45:39.674645 5039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:39.698401 5039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:39.702492 5039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:39.702848 5039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:39.703783 5039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:39.705152 5039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:39.705289 5039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:39.705371 5039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:39.705420 5039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:39.715955 5039 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:38089
I20260620 21:45:39.715973 5167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:38089 every 8 connection(s)
I20260620 21:45:39.716975 5039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:45:39.718906 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 5039
I20260620 21:45:39.719044 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:45:39.719388 1646 external_mini_cluster.cc:1428] Setting key 1486721fd26aa17c7818ea8cbb7574e4
Jun 20 21:45:39 dist-test-slave-5bvn krb5kdc[4664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991939, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:39.732817 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:39.719362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:49305 (local address 127.1.155.190:42559)
0620 21:45:39.719508 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:39.719512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:39.720691 (+ 1179us) server_negotiation.cc:407] Connection header received
0620 21:45:39.722473 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:39.722477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:39.722533 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:39.722612 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:39.723837 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:39.724370 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:39.725563 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:39.725791 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:39.725928 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:39.728539 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:39.728543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:39.728545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:39.728573 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:39.730232 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:39.730860 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:39.730863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:39.730865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:39.730947 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:39.731382 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:39.731386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:39.731387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:39.731589 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:39.731682 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:39.732473 (+ 791us) server_negotiation.cc:299] Negotiation successful
0620 21:45:39.732637 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:45:39.733759 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:39.719889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42559 (local address 127.1.155.131:49305)
0620 21:45:39.720517 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:39.720572 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:39.722246 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:39.722801 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:39.722808 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:39.723180 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:39.723685 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:39.723698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:39.724502 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:39.724506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:39.725429 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:39.725438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:39.725971 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:39.725976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:39.726159 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:39.726763 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:39.726785 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:39.728335 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:39.730366 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:39.730379 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:39.730398 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:39.730724 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:39.731072 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:39.731075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:39.731077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:39.731242 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:39.731733 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:39.731739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:39.732083 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 21:45:39.732350 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 21:45:39.732719 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":145,"threads_started":1}
I20260620 21:45:39.735024 5168 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42559
I20260620 21:45:39.735285 5168 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:39.735795 5168 heartbeater.cc:511] Master 127.1.155.190:42559 requested a full tablet report, sending...
I20260620 21:45:39.736853 4708 ts_manager.cc:194] Registered new tserver with Master: 7dbb8e724c5d4197b7ee0f41f11a22a3 (127.1.155.131:38089)
I20260620 21:45:39.737504 4708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:49305
I20260620 21:45:39.746168 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:45:39 dist-test-slave-5bvn krb5kdc[4664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991939, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 21:45:39 dist-test-slave-5bvn krb5kdc[4664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991939, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:39.773443 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:39.764266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.1.155.190:42559)
0620 21:45:39.764442 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:39.764445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:39.764551 (+ 106us) server_negotiation.cc:407] Connection header received
0620 21:45:39.764658 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:39.764660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:39.764706 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:39.764787 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:39.765464 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:39.766109 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:39.766942 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:39.767262 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:39.767358 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:39.770145 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:39.770148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:39.770150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:39.770178 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:39.771796 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:39.772226 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:39.772229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:39.772230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:39.772306 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:39.772629 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:39.772633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:39.772634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:39.772863 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:39.772966 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:39.773223 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 21:45:39.773321 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 21:45:39.773973 4708 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49046
I20260620 21:45:39.775002 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 4771
I20260620 21:45:39.780804 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 4905
I20260620 21:45:39.786132 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 5039
I20260620 21:45:39.791261 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 4677
2026-06-20T21:45:39Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2172 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:45:39 dist-test-slave-5bvn krb5kdc[5180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:45:39 dist-test-slave-5bvn krb5kdc[5180](info): set up 2 sockets
Jun 20 21:45:39 dist-test-slave-5bvn krb5kdc[5180](info): commencing operation
krb5kdc: starting...
W20260620 21:45:40.846038 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:45:40 dist-test-slave-5bvn krb5kdc[5180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:45:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:45:40.967252 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:39455
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:39455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:41.079710 5193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:41.080065 5193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:41.080138 5193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:41.084398 5193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:41.084507 5193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:41.084548 5193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:41.084596 5193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:41.084631 5193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:45:41.090250 5193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:39455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:39455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:41.091463 5193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:41.092525 5193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:41.101354 5201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:41.101346 5199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:41.101605 5193 server_base.cc:1034] running on GCE node
W20260620 21:45:41.101346 5198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:41.102326 5193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:41.103464 5193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:41.104650 5193 hybrid_clock.cc:648] HybridClock initialized: now 1781991941104618 us; error 49 us; skew 500 ppm
Jun 20 21:45:41 dist-test-slave-5bvn krb5kdc[5180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991941, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:41.107784 5193 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:45:41.108723 5193 webserver.cc:460] Webserver started at http://127.1.155.190:44459/ using document root <none> and password file <none>
I20260620 21:45:41.109290 5193 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:41.109354 5193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:41.109531 5193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:41.111729 5193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "3294b9ce928f4b7187dfc70b863a8a4e"
format_stamp: "Formatted at 2026-06-20 21:45:41 on dist-test-slave-5bvn"
server_key: "723ebfa07f59cb3a6b8a5c8c72e93594"
server_key_iv: "08de530b9001e9033c6405f278982f40"
server_key_version: "encryptionkey@0"
I20260620 21:45:41.112282 5193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "3294b9ce928f4b7187dfc70b863a8a4e"
format_stamp: "Formatted at 2026-06-20 21:45:41 on dist-test-slave-5bvn"
server_key: "723ebfa07f59cb3a6b8a5c8c72e93594"
server_key_iv: "08de530b9001e9033c6405f278982f40"
server_key_version: "encryptionkey@0"
I20260620 21:45:41.116236 5193 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:45:41.118729 5208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:41.119956 5193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:45:41.120102 5193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "3294b9ce928f4b7187dfc70b863a8a4e"
format_stamp: "Formatted at 2026-06-20 21:45:41 on dist-test-slave-5bvn"
server_key: "723ebfa07f59cb3a6b8a5c8c72e93594"
server_key_iv: "08de530b9001e9033c6405f278982f40"
server_key_version: "encryptionkey@0"
I20260620 21:45:41.120205 5193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:41.129340 5193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:41.132428 5193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:41.132627 5193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:41.139456 5193 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:39455
I20260620 21:45:41.139530 5260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:39455 every 8 connection(s)
I20260620 21:45:41.140393 5193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:41.143036 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:41.143325 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 5193
I20260620 21:45:41.143453 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:41.143728 1646 external_mini_cluster.cc:1428] Setting key 5814958a5573e11041a076a658c31fbe
I20260620 21:45:41.150030 5261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e: Bootstrap starting.
I20260620 21:45:41.152637 5261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e: Neither blocks nor log segments found. Creating new log.
Jun 20 21:45:41 dist-test-slave-5bvn krb5kdc[5180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:41.153540 5261 log.cc:824] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:41.155441 5261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e: No bootstrap required, opened a new log
I20260620 21:45:41.158308 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:41.144962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.1.155.190:39455)
0620 21:45:41.145359 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:41.145376 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:41.145417 (+ 41us) server_negotiation.cc:407] Connection header received
0620 21:45:41.146312 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:41.146335 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:41.146737 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:41.147099 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:41.147733 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.149230 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.150114 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.150758 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.150912 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:41.153622 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:41.153627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:41.153662 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:41.153702 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:41.155456 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:41.155991 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:41.155996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:41.156000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:41.156113 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:41.156392 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:41.156394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:41.156395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:41.156790 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:41.156950 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:41.157232 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 21:45:41.157524 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":105,"threads_started":1}
I20260620 21:45:41.159408 5261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294b9ce928f4b7187dfc70b863a8a4e" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39455 } }
I20260620 21:45:41.159660 5261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:41.159734 5261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3294b9ce928f4b7187dfc70b863a8a4e, State: Initialized, Role: FOLLOWER
I20260620 21:45:41.160192 5261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294b9ce928f4b7187dfc70b863a8a4e" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39455 } }
I20260620 21:45:41.160333 5261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:41.160414 5261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:41.160537 5261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:41.161444 5261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294b9ce928f4b7187dfc70b863a8a4e" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39455 } }
I20260620 21:45:41.161775 5261 leader_election.cc:302] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3294b9ce928f4b7187dfc70b863a8a4e; no voters:
I20260620 21:45:41.162093 5261 leader_election.cc:288] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:41.162211 5266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:41.162441 5266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [term 1 LEADER]: Becoming Leader. State: Replica: 3294b9ce928f4b7187dfc70b863a8a4e, State: Running, Role: LEADER
I20260620 21:45:41.162832 5266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294b9ce928f4b7187dfc70b863a8a4e" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39455 } }
I20260620 21:45:41.163408 5261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:41.164565 5268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3294b9ce928f4b7187dfc70b863a8a4e. Latest consensus state: current_term: 1 leader_uuid: "3294b9ce928f4b7187dfc70b863a8a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294b9ce928f4b7187dfc70b863a8a4e" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39455 } } }
I20260620 21:45:41.164692 5268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:41.165123 5275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:41.165076 5267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3294b9ce928f4b7187dfc70b863a8a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294b9ce928f4b7187dfc70b863a8a4e" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39455 } } }
I20260620 21:45:41.165235 5267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:41.168944 5275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:41.174340 5275 catalog_manager.cc:1347] Generated new cluster ID: 008c25d819a9489d9751387405f1024b
I20260620 21:45:41.174414 5275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:41.198843 5275 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:41.199867 5275 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:41.211436 5275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3294b9ce928f4b7187dfc70b863a8a4e: Generated new TSK 0
I20260620 21:45:41.212330 5275 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:45:41.245803 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39455
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:41.355250 5287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:41.355525 5287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:41.355602 5287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:41.359587 5287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:41.359658 5287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:41.359756 5287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:45:41.365378 5287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:41.366434 5287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:41.367419 5287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:41.376837 5295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:41.376811 5292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:41.376773 5293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:41.377012 5287 server_base.cc:1034] running on GCE node
I20260620 21:45:41.377509 5287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:41.378193 5287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:41.379346 5287 hybrid_clock.cc:648] HybridClock initialized: now 1781991941379339 us; error 37 us; skew 500 ppm
Jun 20 21:45:41 dist-test-slave-5bvn krb5kdc[5180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991941, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:41.382055 5287 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:45:41.382870 5287 webserver.cc:460] Webserver started at http://127.1.155.129:44815/ using document root <none> and password file <none>
I20260620 21:45:41.383400 5287 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:41.383462 5287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:41.383654 5287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:41.385778 5287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "7e059213fbe64e9482a2457f0587926f"
format_stamp: "Formatted at 2026-06-20 21:45:41 on dist-test-slave-5bvn"
server_key: "9d4ccdb15fdf1c5e9baea9ee32355575"
server_key_iv: "5322f3a2dc22473e3a31ed3d80b70c7c"
server_key_version: "encryptionkey@0"
I20260620 21:45:41.386371 5287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "7e059213fbe64e9482a2457f0587926f"
format_stamp: "Formatted at 2026-06-20 21:45:41 on dist-test-slave-5bvn"
server_key: "9d4ccdb15fdf1c5e9baea9ee32355575"
server_key_iv: "5322f3a2dc22473e3a31ed3d80b70c7c"
server_key_version: "encryptionkey@0"
I20260620 21:45:41.389817 5287 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:45:41.392205 5302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:41.393388 5287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 21:45:41.393520 5287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "7e059213fbe64e9482a2457f0587926f"
format_stamp: "Formatted at 2026-06-20 21:45:41 on dist-test-slave-5bvn"
server_key: "9d4ccdb15fdf1c5e9baea9ee32355575"
server_key_iv: "5322f3a2dc22473e3a31ed3d80b70c7c"
server_key_version: "encryptionkey@0"
I20260620 21:45:41.393627 5287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:41.419404 5287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:41.422148 5287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:41.422336 5287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:41.422885 5287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:41.423700 5287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:41.423767 5287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:41.423861 5287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:41.423900 5287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:41.432255 5287 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:40439
I20260620 21:45:41.432332 5415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:40439 every 8 connection(s)
I20260620 21:45:41.433128 5287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
Jun 20 21:45:41 dist-test-slave-5bvn krb5kdc[5180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991941, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:41.442358 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 5287
I20260620 21:45:41.442492 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:45:41.442764 1646 external_mini_cluster.cc:1428] Setting key b766e79b75f53674b18483c4181f7f5f
I20260620 21:45:41.446676 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:41.434913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:48043 (local address 127.1.155.190:39455)
0620 21:45:41.435062 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:41.435066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:41.435854 (+ 788us) server_negotiation.cc:407] Connection header received
0620 21:45:41.437222 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:41.437225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:41.437281 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:41.437383 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:41.438520 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.439062 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.440209 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.440444 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.440550 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:41.442983 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:41.442986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:41.442988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:41.443011 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:41.444458 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:41.445046 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:41.445049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:41.445050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:41.445119 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:41.445507 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:41.445512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:41.445515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:41.445753 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:41.445903 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:41.446341 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 21:45:41.446480 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 21:45:41.447352 5418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:41.435263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39455 (local address 127.1.155.129:48043)
0620 21:45:41.435697 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:41.435748 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:41.437017 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:41.437566 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:41.437573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:41.437928 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:41.438364 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:41.438377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.439183 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.439186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:41.440064 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:41.440073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.440584 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.440588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:41.440774 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:41.441313 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:41.441329 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:41.442809 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:41.444570 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:41.444579 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:41.444597 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:41.444952 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:41.445202 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:41.445208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:41.445213 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:41.445391 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:41.445899 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:41.445904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:41.446120 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 21:45:41.446271 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 21:45:41.446506 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":121,"threads_started":1}
I20260620 21:45:41.448515 5416 heartbeater.cc:347] Connected to a master server at 127.1.155.190:39455
I20260620 21:45:41.448748 5416 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:41.449188 5416 heartbeater.cc:511] Master 127.1.155.190:39455 requested a full tablet report, sending...
I20260620 21:45:41.450558 5225 ts_manager.cc:194] Registered new tserver with Master: 7e059213fbe64e9482a2457f0587926f (127.1.155.129:40439)
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
I20260620 21:45:41.451697 5225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:48043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:45:41.468986 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39455
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:41.578961 5421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:41.579196 5421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:41.579239 5421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:41.583292 5421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:41.583344 5421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:41.583448 5421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:45:41.588587 5421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:41.589593 5421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:41.590583 5421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:41.600526 5426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:41.600579 5429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:41.600649 5421 server_base.cc:1034] running on GCE node
W20260620 21:45:41.600591 5427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:41.601120 5421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:41.601931 5421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:41.603127 5421 hybrid_clock.cc:648] HybridClock initialized: now 1781991941603110 us; error 29 us; skew 500 ppm
Jun 20 21:45:41 dist-test-slave-5bvn krb5kdc[5180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991941, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:41.605985 5421 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:45:41.606772 5421 webserver.cc:460] Webserver started at http://127.1.155.130:38445/ using document root <none> and password file <none>
I20260620 21:45:41.607293 5421 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:41.607350 5421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:41.607508 5421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:41.609967 5421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "4aa25c79adff4166a8d14f8a18e142b4"
format_stamp: "Formatted at 2026-06-20 21:45:41 on dist-test-slave-5bvn"
server_key: "4d46e929348dc598ac72bdcdda0c0cc5"
server_key_iv: "396a36f5fa51456fa6aab0764c96566e"
server_key_version: "encryptionkey@0"
I20260620 21:45:41.610699 5421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "4aa25c79adff4166a8d14f8a18e142b4"
format_stamp: "Formatted at 2026-06-20 21:45:41 on dist-test-slave-5bvn"
server_key: "4d46e929348dc598ac72bdcdda0c0cc5"
server_key_iv: "396a36f5fa51456fa6aab0764c96566e"
server_key_version: "encryptionkey@0"
I20260620 21:45:41.614948 5421 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:45:41.618332 5436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:41.623332 5421 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260620 21:45:41.623502 5421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "4aa25c79adff4166a8d14f8a18e142b4"
format_stamp: "Formatted at 2026-06-20 21:45:41 on dist-test-slave-5bvn"
server_key: "4d46e929348dc598ac72bdcdda0c0cc5"
server_key_iv: "396a36f5fa51456fa6aab0764c96566e"
server_key_version: "encryptionkey@0"
I20260620 21:45:41.623637 5421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:41.648487 5421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:41.651901 5421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:41.652096 5421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:41.652701 5421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:41.653517 5421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:41.653582 5421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:41.653647 5421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:41.653693 5421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:41.664100 5421 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:32881
I20260620 21:45:41.664116 5549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:32881 every 8 connection(s)
I20260620 21:45:41.665207 5421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
Jun 20 21:45:41 dist-test-slave-5bvn krb5kdc[5180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991941, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:41.675338 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 5421
I20260620 21:45:41.675482 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:45:41.675830 1646 external_mini_cluster.cc:1428] Setting key 676cc3031ea7efb2865897e7f02626ef
I20260620 21:45:41.678539 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:41.666988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:55581 (local address 127.1.155.190:39455)
0620 21:45:41.667116 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:41.667120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:41.667943 (+ 823us) server_negotiation.cc:407] Connection header received
0620 21:45:41.669230 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:41.669233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:41.669278 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:41.669355 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:41.670540 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.671058 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.672137 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.672350 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.672437 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:41.674861 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:41.674864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:41.674866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:41.674888 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:41.676461 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:41.676926 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:41.676930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:41.676935 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:41.677040 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:41.677451 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:41.677453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:41.677454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:41.677622 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:41.677696 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:41.678321 (+ 625us) server_negotiation.cc:299] Negotiation successful
0620 21:45:41.678425 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:45:41.679492 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:41.667374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39455 (local address 127.1.155.130:55581)
0620 21:45:41.667793 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:41.667833 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:41.669045 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:41.669507 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:41.669514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:41.669947 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:41.670408 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:41.670420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.671169 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.671172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:41.672030 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:41.672039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.672475 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.672477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:41.672605 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:41.673189 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:41.673205 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:41.674686 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:41.676580 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:41.676585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:41.676596 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:41.676830 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:41.677147 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:41.677151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:41.677152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:41.677318 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:41.677720 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:41.677729 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:41.678073 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 21:45:41.678281 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 21:45:41.678645 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":114,"threads_started":1}
I20260620 21:45:41.680517 5550 heartbeater.cc:347] Connected to a master server at 127.1.155.190:39455
I20260620 21:45:41.680747 5550 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:41.681298 5550 heartbeater.cc:511] Master 127.1.155.190:39455 requested a full tablet report, sending...
I20260620 21:45:41.682485 5225 ts_manager.cc:194] Registered new tserver with Master: 4aa25c79adff4166a8d14f8a18e142b4 (127.1.155.130:32881)
I20260620 21:45:41.683065 5225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:55581
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:45:41.705140 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39455
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:41.818141 5555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:41.818410 5555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:41.818454 5555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:41.822570 5555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:41.822625 5555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:41.822685 5555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:45:41.827991 5555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:41.829013 5555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:41.830067 5555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:41.839273 5563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:41.839394 5555 server_base.cc:1034] running on GCE node
W20260620 21:45:41.839279 5560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:41.839283 5561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:41.839951 5555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:41.840551 5555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:41.841724 5555 hybrid_clock.cc:648] HybridClock initialized: now 1781991941841687 us; error 48 us; skew 500 ppm
Jun 20 21:45:41 dist-test-slave-5bvn krb5kdc[5180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991941, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:41.844597 5555 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:45:41.845429 5555 webserver.cc:460] Webserver started at http://127.1.155.131:42441/ using document root <none> and password file <none>
I20260620 21:45:41.846040 5555 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:41.846125 5555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:41.846340 5555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:41.848278 5555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "49f30438fd864e8cb0e184e21c4be47a"
format_stamp: "Formatted at 2026-06-20 21:45:41 on dist-test-slave-5bvn"
server_key: "afc9b1eb66be2b629ccfd96258528fb5"
server_key_iv: "f35e611c6492a2380f35628060be84d2"
server_key_version: "encryptionkey@0"
I20260620 21:45:41.848827 5555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "49f30438fd864e8cb0e184e21c4be47a"
format_stamp: "Formatted at 2026-06-20 21:45:41 on dist-test-slave-5bvn"
server_key: "afc9b1eb66be2b629ccfd96258528fb5"
server_key_iv: "f35e611c6492a2380f35628060be84d2"
server_key_version: "encryptionkey@0"
I20260620 21:45:41.852425 5555 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:45:41.854908 5570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:41.856132 5555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:45:41.856360 5555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "49f30438fd864e8cb0e184e21c4be47a"
format_stamp: "Formatted at 2026-06-20 21:45:41 on dist-test-slave-5bvn"
server_key: "afc9b1eb66be2b629ccfd96258528fb5"
server_key_iv: "f35e611c6492a2380f35628060be84d2"
server_key_version: "encryptionkey@0"
I20260620 21:45:41.856509 5555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:41.885880 5555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:41.888833 5555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:41.889036 5555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:41.889616 5555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:41.890693 5555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:41.890789 5555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:41.890859 5555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:41.890915 5555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:41.900734 5555 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:42565
I20260620 21:45:41.900789 5683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:42565 every 8 connection(s)
I20260620 21:45:41.901687 5555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
Jun 20 21:45:41 dist-test-slave-5bvn krb5kdc[5180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991941, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:41.912029 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 5555
I20260620 21:45:41.912168 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:45:41.912516 1646 external_mini_cluster.cc:1428] Setting key 85e39bc14c940148b6e5f3487278a59f
I20260620 21:45:41.915037 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:41.903547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:41935 (local address 127.1.155.190:39455)
0620 21:45:41.903690 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:41.903693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:41.904558 (+ 865us) server_negotiation.cc:407] Connection header received
0620 21:45:41.905941 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:41.905944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:41.905996 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:41.906070 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:41.907144 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.907680 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.908747 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.908977 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.909058 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:41.911403 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:41.911406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:41.911408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:41.911431 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:41.912947 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:41.913430 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:41.913436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:41.913438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:41.913521 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:41.913878 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:41.913881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:41.913883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:41.914105 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:41.914209 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:41.914753 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 21:45:41.914866 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:45:41.915802 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:41.903909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39455 (local address 127.1.155.131:41935)
0620 21:45:41.904416 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:41.904463 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:41.905685 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:41.906208 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:41.906214 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:41.906561 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:41.907013 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:41.907026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.907785 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.907788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:41.908644 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:41.908651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.909098 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.909100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:41.909227 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:41.909873 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:41.909910 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:41.911261 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:41.913060 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:41.913067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:41.913080 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:41.913328 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:41.913609 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:41.913614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:41.913616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:41.913761 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:41.914205 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:41.914211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:41.914507 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 21:45:41.914714 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 21:45:41.915031 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":125,"threads_started":1}
I20260620 21:45:41.916883 5684 heartbeater.cc:347] Connected to a master server at 127.1.155.190:39455
I20260620 21:45:41.917122 5684 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:41.917580 5684 heartbeater.cc:511] Master 127.1.155.190:39455 requested a full tablet report, sending...
I20260620 21:45:41.918579 5225 ts_manager.cc:194] Registered new tserver with Master: 49f30438fd864e8cb0e184e21c4be47a (127.1.155.131:42565)
I20260620 21:45:41.919100 5225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:41935
I20260620 21:45:41.926090 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:45:41 dist-test-slave-5bvn krb5kdc[5180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:45:41 dist-test-slave-5bvn krb5kdc[5180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.129@KRBTEST.COM
I20260620 21:45:41.951210 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:41.942238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50648 (local address 127.1.155.129:40439)
0620 21:45:41.942544 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:41.942559 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:41.942582 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:45:41.942631 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:41.942636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:41.942819 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:41.942937 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:41.943577 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.944358 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.945161 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.945531 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.945613 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:41.947849 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:41.947858 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:41.947870 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:41.947892 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:41.949458 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:41.949927 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:41.949932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:41.949933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:41.950009 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:41.950269 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:41.950274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:41.950275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:41.950515 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:41.950720 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:41.950901 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 21:45:41.951044 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":101,"threads_started":1}
Jun 20 21:45:41 dist-test-slave-5bvn krb5kdc[5180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:41.961766 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:41.954001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.1.155.190:39455)
0620 21:45:41.954149 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:41.954152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:41.954165 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:45:41.954247 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:41.954249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:41.954289 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:41.954355 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:41.954901 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.955397 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.956123 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:41.956318 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:41.956387 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:41.958669 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:41.958673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:41.958675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:41.958698 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:41.960188 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:41.960570 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:41.960575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:41.960577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:41.960655 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:41.960900 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:41.960905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:41.960906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:41.961117 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:41.961255 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:41.961444 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 21:45:41.961561 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 21:45:41.962194 5225 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46786
I20260620 21:45:41.962841 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 5287
I20260620 21:45:41.968746 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 5421
I20260620 21:45:41.974134 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 5555
I20260620 21:45:41.979697 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 5193
2026-06-20T21:45:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2188 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:45:42 dist-test-slave-5bvn krb5kdc[5696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:45:42 dist-test-slave-5bvn krb5kdc[5696](info): set up 2 sockets
Jun 20 21:45:42 dist-test-slave-5bvn krb5kdc[5696](info): commencing operation
krb5kdc: starting...
W20260620 21:45:43.046015 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:45:43 dist-test-slave-5bvn krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:45:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:45:43.167088 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:42605
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:42605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:43.274657 5709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:43.274914 5709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:43.274957 5709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:43.278985 5709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:43.279035 5709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:43.279055 5709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:43.279072 5709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:43.279116 5709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:45:43.284302 5709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:42605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:42605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:43.285272 5709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:43.286271 5709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:43.294173 5714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:43.294140 5717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:43.294155 5715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:43.294466 5709 server_base.cc:1034] running on GCE node
I20260620 21:45:43.294979 5709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:43.295994 5709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:43.297175 5709 hybrid_clock.cc:648] HybridClock initialized: now 1781991943297148 us; error 40 us; skew 500 ppm
Jun 20 21:45:43 dist-test-slave-5bvn krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:43.306133 5709 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:45:43.306928 5709 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:43.307022 5709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:43.307243 5709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:43.309252 5709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "bc29f89b568942d4b8dffdcfde1ae133"
format_stamp: "Formatted at 2026-06-20 21:45:43 on dist-test-slave-5bvn"
server_key: "15ef7b6d50b3df1e790406c52825fa5e"
server_key_iv: "3bf7f6ccb2fbdd6508aa4096e6bf3d64"
server_key_version: "encryptionkey@0"
I20260620 21:45:43.309803 5709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "bc29f89b568942d4b8dffdcfde1ae133"
format_stamp: "Formatted at 2026-06-20 21:45:43 on dist-test-slave-5bvn"
server_key: "15ef7b6d50b3df1e790406c52825fa5e"
server_key_iv: "3bf7f6ccb2fbdd6508aa4096e6bf3d64"
server_key_version: "encryptionkey@0"
I20260620 21:45:43.313416 5709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:45:43.315901 5723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:43.317157 5709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:45:43.317306 5709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "bc29f89b568942d4b8dffdcfde1ae133"
format_stamp: "Formatted at 2026-06-20 21:45:43 on dist-test-slave-5bvn"
server_key: "15ef7b6d50b3df1e790406c52825fa5e"
server_key_iv: "3bf7f6ccb2fbdd6508aa4096e6bf3d64"
server_key_version: "encryptionkey@0"
I20260620 21:45:43.317433 5709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:43.343281 5709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:43.346624 5709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:43.346825 5709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:43.354200 5709 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:42605
I20260620 21:45:43.354454 5775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:42605 every 8 connection(s)
I20260620 21:45:43.354905 5709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:43.357532 5776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:43.362802 5776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133: Bootstrap starting.
I20260620 21:45:43.363461 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 5709
I20260620 21:45:43.363590 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:43.363863 1646 external_mini_cluster.cc:1428] Setting key 3fc551477a99f534532e2cef020fd074
I20260620 21:45:43.365576 5776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:43.366465 5776 log.cc:824] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:43.368585 5776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133: No bootstrap required, opened a new log
I20260620 21:45:43.372612 5776 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc29f89b568942d4b8dffdcfde1ae133" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42605 } }
I20260620 21:45:43.372897 5776 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:43.372967 5776 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc29f89b568942d4b8dffdcfde1ae133, State: Initialized, Role: FOLLOWER
Jun 20 21:45:43 dist-test-slave-5bvn krb5kdc[5696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:43.373392 5776 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc29f89b568942d4b8dffdcfde1ae133" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42605 } }
I20260620 21:45:43.373544 5776 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:43.373631 5776 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:43.373731 5776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:43.374910 5776 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc29f89b568942d4b8dffdcfde1ae133" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42605 } }
I20260620 21:45:43.375272 5776 leader_election.cc:302] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc29f89b568942d4b8dffdcfde1ae133; no voters:
I20260620 21:45:43.375597 5776 leader_election.cc:288] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:43.375743 5781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:43.376000 5781 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [term 1 LEADER]: Becoming Leader. State: Replica: bc29f89b568942d4b8dffdcfde1ae133, State: Running, Role: LEADER
I20260620 21:45:43.376394 5781 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc29f89b568942d4b8dffdcfde1ae133" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42605 } }
I20260620 21:45:43.376927 5776 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:43.378186 5782 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc29f89b568942d4b8dffdcfde1ae133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc29f89b568942d4b8dffdcfde1ae133" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42605 } } }
I20260620 21:45:43.378319 5782 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:43.378706 5783 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc29f89b568942d4b8dffdcfde1ae133. Latest consensus state: current_term: 1 leader_uuid: "bc29f89b568942d4b8dffdcfde1ae133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc29f89b568942d4b8dffdcfde1ae133" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42605 } } }
I20260620 21:45:43.378872 5783 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133 [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:43.379849 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:43.365143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.1.155.190:42605)
0620 21:45:43.365568 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:43.365589 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:43.365634 (+ 45us) server_negotiation.cc:407] Connection header received
0620 21:45:43.366517 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:43.366541 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:43.366943 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:43.367285 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:43.367912 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:43.369332 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:43.370241 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:43.370929 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:43.371101 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:43.373708 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:43.373714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:43.373731 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:43.373764 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:43.375787 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:43.376305 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:43.376311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:43.376316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:43.376442 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:43.376795 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:43.376799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:43.376801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:43.377309 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:43.377505 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:43.378223 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 21:45:43.378545 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
W20260620 21:45:43.381721 5796 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:45:43.381805 5796 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:45:43.382063 5790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:43.382787 5790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:43.388392 5790 catalog_manager.cc:1347] Generated new cluster ID: 0491a302673347368a31d16715db3149
I20260620 21:45:43.388456 5790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:43.409185 5790 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:43.410219 5790 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:43.416173 5790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133: Generated new TSK 0
I20260620 21:45:43.417021 5790 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:45:43.450506 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42605
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:43.557590 5802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:43.557826 5802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:43.557888 5802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:43.561781 5802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:43.561852 5802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:43.561928 5802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:45:43.566987 5802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:43.567889 5802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:43.568801 5802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:43.577637 5808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:43.577631 5810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:43.577629 5807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:43.578032 5802 server_base.cc:1034] running on GCE node
I20260620 21:45:43.578491 5802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:43.579113 5802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:43.580307 5802 hybrid_clock.cc:648] HybridClock initialized: now 1781991943580297 us; error 49 us; skew 500 ppm
Jun 20 21:45:43 dist-test-slave-5bvn krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:43.583040 5802 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:45:43.583739 5802 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:43.583802 5802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:43.583957 5802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:43.586004 5802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "fb946fbf90da4cb59b723f4b8cd53fdd"
format_stamp: "Formatted at 2026-06-20 21:45:43 on dist-test-slave-5bvn"
server_key: "0dbe70df3c725ec82a47491a6071376f"
server_key_iv: "c210077b5a0cb33e67ce8734fd1432f0"
server_key_version: "encryptionkey@0"
I20260620 21:45:43.586491 5802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "fb946fbf90da4cb59b723f4b8cd53fdd"
format_stamp: "Formatted at 2026-06-20 21:45:43 on dist-test-slave-5bvn"
server_key: "0dbe70df3c725ec82a47491a6071376f"
server_key_iv: "c210077b5a0cb33e67ce8734fd1432f0"
server_key_version: "encryptionkey@0"
I20260620 21:45:43.590540 5802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 21:45:43.592976 5816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:43.594269 5802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:45:43.594398 5802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "fb946fbf90da4cb59b723f4b8cd53fdd"
format_stamp: "Formatted at 2026-06-20 21:45:43 on dist-test-slave-5bvn"
server_key: "0dbe70df3c725ec82a47491a6071376f"
server_key_iv: "c210077b5a0cb33e67ce8734fd1432f0"
server_key_version: "encryptionkey@0"
I20260620 21:45:43.594511 5802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:43.637633 5802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:43.640555 5802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:43.640759 5802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:43.641355 5802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:43.642231 5802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:43.642279 5802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:43.642346 5802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:43.642385 5802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:43.650868 5802 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:45315
I20260620 21:45:43.650895 5929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:45315 every 8 connection(s)
I20260620 21:45:43.651424 5802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:45:43.656502 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 5802
I20260620 21:45:43.656606 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:45:43.656927 1646 external_mini_cluster.cc:1428] Setting key 27945af5165874e2006d63304a5b1d45
Jun 20 21:45:43 dist-test-slave-5bvn krb5kdc[5696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:43.666215 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:43.653125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:52299 (local address 127.1.155.190:42605)
0620 21:45:43.653257 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:43.653260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:43.654171 (+ 911us) server_negotiation.cc:407] Connection header received
0620 21:45:43.655458 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:43.655460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:43.655506 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:43.655582 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:43.656810 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:43.657543 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:43.658966 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:43.659388 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:43.659514 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:43.662333 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:43.662338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:43.662342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:43.662376 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:43.663791 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:43.664474 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:43.664477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:43.664478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:43.664560 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:43.664989 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:43.664992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:43.664993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:43.665207 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:43.665298 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:43.665992 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 21:45:43.666094 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
I20260620 21:45:43.667207 5932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:43.653520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42605 (local address 127.1.155.129:52299)
0620 21:45:43.654025 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:43.654069 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:43.655273 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:43.655763 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:43.655770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:43.656109 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:43.656656 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:43.656674 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:43.657670 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:43.657674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:43.658840 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:43.658851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:43.659530 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:43.659533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:43.659672 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:43.660312 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:43.660329 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:43.662129 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:43.663919 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:43.663925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:43.663941 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:43.664326 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:43.664658 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:43.664661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:43.664663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:43.664853 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:43.665335 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:43.665341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:43.665679 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 21:45:43.665928 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 21:45:43.666271 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":98,"threads_started":1}
I20260620 21:45:43.668560 5930 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
I20260620 21:45:43.668845 5930 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:43.669395 5930 heartbeater.cc:511] Master 127.1.155.190:42605 requested a full tablet report, sending...
I20260620 21:45:43.671247 5740 ts_manager.cc:194] Registered new tserver with Master: fb946fbf90da4cb59b723f4b8cd53fdd (127.1.155.129:45315)
I20260620 21:45:43.672928 5740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:52299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:45:43.685271 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42605
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:43.795027 5935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:43.795322 5935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:43.795369 5935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:43.799371 5935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:43.799428 5935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:43.799489 5935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:45:43.804636 5935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:43.805624 5935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:43.806651 5935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:43.815996 5943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:43.815940 5941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:43.815940 5940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:43.816222 5935 server_base.cc:1034] running on GCE node
I20260620 21:45:43.816707 5935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:43.817358 5935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:43.818531 5935 hybrid_clock.cc:648] HybridClock initialized: now 1781991943818517 us; error 27 us; skew 500 ppm
Jun 20 21:45:43 dist-test-slave-5bvn krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:43.821223 5935 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:45:43.821982 5935 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:43.822048 5935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:43.822216 5935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:43.824093 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "74916471773d44fdab143f409fa2eaeb"
format_stamp: "Formatted at 2026-06-20 21:45:43 on dist-test-slave-5bvn"
server_key: "8c5d6ed5612a0f1c1f2b6939c15eb433"
server_key_iv: "20f5481285f40953699e61c85b88690f"
server_key_version: "encryptionkey@0"
I20260620 21:45:43.824586 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "74916471773d44fdab143f409fa2eaeb"
format_stamp: "Formatted at 2026-06-20 21:45:43 on dist-test-slave-5bvn"
server_key: "8c5d6ed5612a0f1c1f2b6939c15eb433"
server_key_iv: "20f5481285f40953699e61c85b88690f"
server_key_version: "encryptionkey@0"
I20260620 21:45:43.828162 5935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:45:43.830518 5949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:43.831835 5935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:45:43.831966 5935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "74916471773d44fdab143f409fa2eaeb"
format_stamp: "Formatted at 2026-06-20 21:45:43 on dist-test-slave-5bvn"
server_key: "8c5d6ed5612a0f1c1f2b6939c15eb433"
server_key_iv: "20f5481285f40953699e61c85b88690f"
server_key_version: "encryptionkey@0"
I20260620 21:45:43.832073 5935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:43.844262 5935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:43.847199 5935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:43.847429 5935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:43.847980 5935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:43.848768 5935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:43.848821 5935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:43.848902 5935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:43.848932 5935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:43.857570 5935 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:34961
I20260620 21:45:43.857591 6062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:34961 every 8 connection(s)
I20260620 21:45:43.858144 5935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:45:43.861429 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 5935
I20260620 21:45:43.861567 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:45:43.861917 1646 external_mini_cluster.cc:1428] Setting key a67744ff4b00253635014313eb749e19
Jun 20 21:45:43 dist-test-slave-5bvn krb5kdc[5696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
I20260620 21:45:43.872565 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:43.859818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:44429 (local address 127.1.155.190:42605)
0620 21:45:43.859952 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:43.859955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:43.860699 (+ 744us) server_negotiation.cc:407] Connection header received
0620 21:45:43.862422 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:43.862425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:43.862485 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:43.862587 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:43.863875 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:43.864405 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:43.865682 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:43.865973 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:43.866084 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:43.868846 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:43.868851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:43.868855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:43.868885 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:43.870606 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:43.871108 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:43.871114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:43.871119 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:43.871212 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:43.871555 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:43.871558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:43.871559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:43.871733 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:43.871823 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:43.872341 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 21:45:43.872445 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:45:43.873436 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:43.860130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42605 (local address 127.1.155.130:44429)
0620 21:45:43.860555 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:43.860596 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:43.862155 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:43.862761 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:43.862768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:43.863099 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:43.863712 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:43.863736 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:43.864550 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:43.864556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:43.865554 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:43.865563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:43.866116 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:43.866120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:43.866293 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:43.866875 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:43.866891 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:43.868656 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:43.870734 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:43.870738 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:43.870750 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:43.871011 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:43.871333 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:43.871335 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:43.871337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:43.871466 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:43.871835 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:43.871840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:43.872059 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 21:45:43.872210 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 21:45:43.872535 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
I20260620 21:45:43.874657 6063 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42605
I20260620 21:45:43.874903 6063 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:43.875438 6063 heartbeater.cc:511] Master 127.1.155.190:42605 requested a full tablet report, sending...
I20260620 21:45:43.876539 5740 ts_manager.cc:194] Registered new tserver with Master: 74916471773d44fdab143f409fa2eaeb (127.1.155.130:34961)
I20260620 21:45:43.877381 5740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:44429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:45:43.889209 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42605
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:43.998883 6068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:43.999169 6068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:43.999250 6068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:44.003450 6068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:44.003521 6068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:44.003587 6068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:45:44.010793 6068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:44.011878 6068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:44.012827 6068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:44.022156 6076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:44.022122 6073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:44.022138 6074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:44.022384 6068 server_base.cc:1034] running on GCE node
I20260620 21:45:44.022910 6068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:44.023592 6068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:44.024770 6068 hybrid_clock.cc:648] HybridClock initialized: now 1781991944024751 us; error 30 us; skew 500 ppm
Jun 20 21:45:44 dist-test-slave-5bvn krb5kdc[5696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991944, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:44.027709 6068 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:45:44.028393 6068 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:44.028458 6068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:44.028620 6068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:44.030614 6068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "64316452fcad4c9cbe6cfb813692bba0"
format_stamp: "Formatted at 2026-06-20 21:45:44 on dist-test-slave-5bvn"
server_key: "138dc05fac0d33e6192633c6752d285f"
server_key_iv: "616f650002617b968943efb4c0993fac"
server_key_version: "encryptionkey@0"
I20260620 21:45:44.031108 6068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "64316452fcad4c9cbe6cfb813692bba0"
format_stamp: "Formatted at 2026-06-20 21:45:44 on dist-test-slave-5bvn"
server_key: "138dc05fac0d33e6192633c6752d285f"
server_key_iv: "616f650002617b968943efb4c0993fac"
server_key_version: "encryptionkey@0"
I20260620 21:45:44.034643 6068 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:45:44.037091 6082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:44.038342 6068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:45:44.038471 6068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "64316452fcad4c9cbe6cfb813692bba0"
format_stamp: "Formatted at 2026-06-20 21:45:44 on dist-test-slave-5bvn"
server_key: "138dc05fac0d33e6192633c6752d285f"
server_key_iv: "616f650002617b968943efb4c0993fac"
server_key_version: "encryptionkey@0"
I20260620 21:45:44.038579 6068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:44.058853 6068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:44.061605 6068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:44.061808 6068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:44.062408 6068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:44.063210 6068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:44.063277 6068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:44.063344 6068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:44.063388 6068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:44.071728 6068 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:44927
I20260620 21:45:44.071750 6195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:44927 every 8 connection(s)
I20260620 21:45:44.072306 6068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:45:44.075460 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 6068
I20260620 21:45:44.075594 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:45:44.075873 1646 external_mini_cluster.cc:1428] Setting key 39a7ea75862719cc330c19ec5f070275
Jun 20 21:45:44 dist-test-slave-5bvn krb5kdc[5696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991944, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:44.085942 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.074053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:42845 (local address 127.1.155.190:42605)
0620 21:45:44.074202 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:44.074207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:44.075064 (+ 857us) server_negotiation.cc:407] Connection header received
0620 21:45:44.076404 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:44.076407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:44.076464 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:44.076545 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:44.078018 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.078629 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.079827 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.080043 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.080123 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:44.082634 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:44.082637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:44.082640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:44.082662 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:44.083953 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:44.084513 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:44.084516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:44.084517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:44.084589 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:44.084865 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:44.084867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:44.084868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:44.085034 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:44.085107 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:44.085637 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 21:45:44.085753 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:45:44.086723 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.074479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42605 (local address 127.1.155.131:42845)
0620 21:45:44.074919 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:44.074959 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:44.076204 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:44.076727 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:44.076733 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:44.077175 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:44.077890 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:44.077911 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.078728 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.078730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:44.079713 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:44.079722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.080163 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.080165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:44.080326 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:44.080915 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:44.080931 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:44.082485 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:44.084090 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:44.084094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:44.084106 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:44.084410 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:44.084660 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:44.084662 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:44.084664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:44.084787 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:44.085112 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:44.085116 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:44.085390 (+ 274us) client_negotiation.cc:769] Sending connection context
0620 21:45:44.085537 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 21:45:44.085778 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":123,"threads_started":1}
I20260620 21:45:44.087940 6196 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42605
I20260620 21:45:44.088189 6196 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:44.088747 6196 heartbeater.cc:511] Master 127.1.155.190:42605 requested a full tablet report, sending...
I20260620 21:45:44.089869 5740 ts_manager.cc:194] Registered new tserver with Master: 64316452fcad4c9cbe6cfb813692bba0 (127.1.155.131:44927)
I20260620 21:45:44.090448 5740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:42845
I20260620 21:45:44.102090 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:45:44.111135 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.104093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54592 (local address 127.1.155.190:42605)
0620 21:45:44.104222 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:44.104226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:44.104341 (+ 115us) server_negotiation.cc:407] Connection header received
0620 21:45:44.104440 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:44.104442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:44.104484 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:44.104563 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:44.105274 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.105799 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.106691 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.106911 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.106990 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:44.108260 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:44.108263 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:44.108264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:44.108287 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:44.109675 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:44.110114 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:44.110116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:44.110118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:44.110190 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:44.110433 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:44.110435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:44.110436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:44.110639 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:44.110752 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:44.110926 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 21:45:44.111016 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:45:44.114082 5740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:45:44.117049 5740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:44.131901 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.125589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:45315 (local address 127.0.0.1:57182)
0620 21:45:44.126002 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:44.126049 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:44.126177 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:44.127514 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:44.127517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:44.127532 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:44.127620 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:44.127625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.129226 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.129231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:44.130564 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:44.130575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.131377 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.131380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:44.131482 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:44.131516 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 21:45:44.131621 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 21:45:44.131733 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":80,"thread_start_us":114,"threads_started":1}
I20260620 21:45:44.132097 6212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.126545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.1.155.130:34961)
0620 21:45:44.127081 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:44.127092 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:44.127303 (+ 211us) server_negotiation.cc:407] Connection header received
0620 21:45:44.127499 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:44.127504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:44.127670 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:44.127800 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:44.128128 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.128853 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.130595 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.131423 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.131540 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:44.131673 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:44.131839 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 21:45:44.131939 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":98,"threads_started":1}
I20260620 21:45:44.132267 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.126403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.1.155.129:45315)
0620 21:45:44.126867 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:44.126875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:44.126897 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:45:44.126973 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:44.126978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:44.127168 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:44.127338 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:44.127738 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.128962 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.130728 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.131260 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.131379 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:44.131523 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:44.131984 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 21:45:44.132091 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":94,"threads_started":1}
I20260620 21:45:44.131960 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.125640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:44927 (local address 127.0.0.1:54344)
0620 21:45:44.126706 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:44.126734 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:44.126831 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:44.127383 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:44.127391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:44.127417 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:44.127560 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:44.127569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.129087 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.129090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:44.130571 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:44.130576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.131563 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.131566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:44.131652 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:44.131673 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:45:44.131724 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 21:45:44.131819 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":104,"thread_start_us":128,"threads_started":1}
I20260620 21:45:44.132552 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.126043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:34961 (local address 127.0.0.1:32956)
0620 21:45:44.127218 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:44.127233 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:44.127362 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:44.127945 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:44.127947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:44.127960 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:44.128023 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:44.128027 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.128977 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.128983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:44.130476 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:44.130485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.131599 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.131604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:44.131691 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:44.131711 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 21:45:44.131792 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 21:45:44.131850 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"mutex_wait_us":210,"thread_start_us":99,"threads_started":1}
I20260620 21:45:44.133067 6210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.126121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.1.155.131:44927)
0620 21:45:44.126673 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:44.126684 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:44.126817 (+ 133us) server_negotiation.cc:407] Connection header received
0620 21:45:44.126947 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:44.126952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:44.127161 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:44.127737 (+ 576us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:44.127878 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.128962 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.130683 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.131426 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.132069 (+ 643us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:44.132225 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:44.132292 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 21:45:44.132926 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":170,"threads_started":1}
I20260620 21:45:44.135411 5997 tablet_service.cc:1467] Processing CreateTablet for tablet 26d7d283e43f4ca69cc3f952615c2748 (DEFAULT_TABLE table=test-table [id=8a842339825e4768a33f6f272cad79a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:44.135762 6130 tablet_service.cc:1467] Processing CreateTablet for tablet 26d7d283e43f4ca69cc3f952615c2748 (DEFAULT_TABLE table=test-table [id=8a842339825e4768a33f6f272cad79a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:44.135758 5864 tablet_service.cc:1467] Processing CreateTablet for tablet 26d7d283e43f4ca69cc3f952615c2748 (DEFAULT_TABLE table=test-table [id=8a842339825e4768a33f6f272cad79a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:44.136550 5997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d7d283e43f4ca69cc3f952615c2748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:44.136734 5864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d7d283e43f4ca69cc3f952615c2748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:44.136734 6130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d7d283e43f4ca69cc3f952615c2748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:44.144266 6214 tablet_bootstrap.cc:492] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0: Bootstrap starting.
I20260620 21:45:44.145409 6213 tablet_bootstrap.cc:492] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd: Bootstrap starting.
I20260620 21:45:44.145469 6215 tablet_bootstrap.cc:492] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb: Bootstrap starting.
I20260620 21:45:44.147001 6214 tablet_bootstrap.cc:654] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:44.147753 6215 tablet_bootstrap.cc:654] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:44.147753 6213 tablet_bootstrap.cc:654] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:44.148209 6214 log.cc:824] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:44.149093 6215 log.cc:824] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:44.149093 6213 log.cc:824] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:44.150134 6214 tablet_bootstrap.cc:492] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0: No bootstrap required, opened a new log
I20260620 21:45:44.150332 6214 ts_tablet_manager.cc:1397] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:45:44.150707 6213 tablet_bootstrap.cc:492] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd: No bootstrap required, opened a new log
I20260620 21:45:44.150848 6213 ts_tablet_manager.cc:1397] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:45:44.152976 6215 tablet_bootstrap.cc:492] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb: No bootstrap required, opened a new log
I20260620 21:45:44.153177 6215 ts_tablet_manager.cc:1397] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 21:45:44.153589 6214 raft_consensus.cc:348] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb946fbf90da4cb59b723f4b8cd53fdd" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 45315 } } peers { permanent_uuid: "64316452fcad4c9cbe6cfb813692bba0" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44927 } } peers { permanent_uuid: "74916471773d44fdab143f409fa2eaeb" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34961 } }
I20260620 21:45:44.153834 6214 raft_consensus.cc:374] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:44.153923 6214 raft_consensus.cc:729] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64316452fcad4c9cbe6cfb813692bba0, State: Initialized, Role: FOLLOWER
I20260620 21:45:44.154364 6214 consensus_queue.cc:260] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb946fbf90da4cb59b723f4b8cd53fdd" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 45315 } } peers { permanent_uuid: "64316452fcad4c9cbe6cfb813692bba0" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44927 } } peers { permanent_uuid: "74916471773d44fdab143f409fa2eaeb" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34961 } }
I20260620 21:45:44.154415 6213 raft_consensus.cc:348] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb946fbf90da4cb59b723f4b8cd53fdd" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 45315 } } peers { permanent_uuid: "64316452fcad4c9cbe6cfb813692bba0" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44927 } } peers { permanent_uuid: "74916471773d44fdab143f409fa2eaeb" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34961 } }
I20260620 21:45:44.154650 6213 raft_consensus.cc:374] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:44.154696 6213 raft_consensus.cc:729] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb946fbf90da4cb59b723f4b8cd53fdd, State: Initialized, Role: FOLLOWER
I20260620 21:45:44.155120 6213 consensus_queue.cc:260] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb946fbf90da4cb59b723f4b8cd53fdd" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 45315 } } peers { permanent_uuid: "64316452fcad4c9cbe6cfb813692bba0" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44927 } } peers { permanent_uuid: "74916471773d44fdab143f409fa2eaeb" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34961 } }
I20260620 21:45:44.155241 6214 ts_tablet_manager.cc:1428] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260620 21:45:44.155635 6196 heartbeater.cc:503] Master 127.1.155.190:42605 was elected leader, sending a full tablet report...
I20260620 21:45:44.156064 6213 ts_tablet_manager.cc:1428] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 21:45:44.156565 5930 heartbeater.cc:503] Master 127.1.155.190:42605 was elected leader, sending a full tablet report...
I20260620 21:45:44.156857 6215 raft_consensus.cc:348] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb946fbf90da4cb59b723f4b8cd53fdd" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 45315 } } peers { permanent_uuid: "64316452fcad4c9cbe6cfb813692bba0" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44927 } } peers { permanent_uuid: "74916471773d44fdab143f409fa2eaeb" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34961 } }
I20260620 21:45:44.157081 6215 raft_consensus.cc:374] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:44.157146 6215 raft_consensus.cc:729] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74916471773d44fdab143f409fa2eaeb, State: Initialized, Role: FOLLOWER
I20260620 21:45:44.157568 6215 consensus_queue.cc:260] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb946fbf90da4cb59b723f4b8cd53fdd" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 45315 } } peers { permanent_uuid: "64316452fcad4c9cbe6cfb813692bba0" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44927 } } peers { permanent_uuid: "74916471773d44fdab143f409fa2eaeb" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34961 } }
I20260620 21:45:44.158233 6063 heartbeater.cc:503] Master 127.1.155.190:42605 was elected leader, sending a full tablet report...
I20260620 21:45:44.158514 6215 ts_tablet_manager.cc:1428] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 21:45:44.323184 6197 tablet.cc:2369] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:45:44.344061 6221 raft_consensus.cc:482] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:44.344310 6221 raft_consensus.cc:504] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb946fbf90da4cb59b723f4b8cd53fdd" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 45315 } } peers { permanent_uuid: "64316452fcad4c9cbe6cfb813692bba0" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44927 } } peers { permanent_uuid: "74916471773d44fdab143f409fa2eaeb" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34961 } }
I20260620 21:45:44.345443 6221 leader_election.cc:288] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb946fbf90da4cb59b723f4b8cd53fdd (127.1.155.129:45315), 64316452fcad4c9cbe6cfb813692bba0 (127.1.155.131:44927)
I20260620 21:45:44.349329 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.345717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:45315 (local address 127.1.155.130:34021)
0620 21:45:44.345854 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:44.345870 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:44.345981 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:44.346283 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:44.346299 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:44.346322 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:44.346457 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:44.346464 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.347279 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.347282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:44.348217 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:44.348226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.348874 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.348878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:44.349007 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:44.349038 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 21:45:44.349110 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 21:45:44.349190 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":20}
I20260620 21:45:44.349354 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.345800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:34021 (local address 127.1.155.129:45315)
0620 21:45:44.345949 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:44.345954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:44.345971 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:45:44.346104 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:44.346107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:44.346157 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:44.346251 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:44.346594 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.347178 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.348347 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.348753 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.348846 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:44.348885 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:44.349145 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 21:45:44.349216 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 21:45:44.349524 6210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.345840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:54009 (local address 127.1.155.131:44927)
0620 21:45:44.345997 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:44.346002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:44.346315 (+ 313us) server_negotiation.cc:407] Connection header received
0620 21:45:44.346446 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:44.346449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:44.346499 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:44.346590 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:44.346825 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.347352 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.348447 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.348941 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.349068 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:44.349127 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:44.349320 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 21:45:44.349401 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:45:44.349766 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.345724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:44927 (local address 127.1.155.130:54009)
0620 21:45:44.346194 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:44.346209 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:44.346340 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:44.346593 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:44.346596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:44.346612 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:44.346720 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:44.346726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.347457 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.347459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:44.348256 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:44.348264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.349055 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.349058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:44.349175 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:44.349197 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:45:44.349282 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 21:45:44.349344 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":203,"thread_start_us":75,"threads_started":1}
I20260620 21:45:44.350014 5884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d7d283e43f4ca69cc3f952615c2748" candidate_uuid: "74916471773d44fdab143f409fa2eaeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb946fbf90da4cb59b723f4b8cd53fdd" is_pre_election: true
I20260620 21:45:44.350364 5884 raft_consensus.cc:2454] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74916471773d44fdab143f409fa2eaeb in term 0.
I20260620 21:45:44.350427 6150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d7d283e43f4ca69cc3f952615c2748" candidate_uuid: "74916471773d44fdab143f409fa2eaeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64316452fcad4c9cbe6cfb813692bba0" is_pre_election: true
I20260620 21:45:44.350693 6150 raft_consensus.cc:2454] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74916471773d44fdab143f409fa2eaeb in term 0.
I20260620 21:45:44.350896 5951 leader_election.cc:302] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74916471773d44fdab143f409fa2eaeb, fb946fbf90da4cb59b723f4b8cd53fdd; no voters:
I20260620 21:45:44.351171 6221 raft_consensus.cc:2781] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:44.351246 6221 raft_consensus.cc:482] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:44.351277 6221 raft_consensus.cc:3037] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [term 0 FOLLOWER]: Advancing to term 1
W20260620 21:45:44.359184 6064 tablet.cc:2369] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:45:44.375118 6221 raft_consensus.cc:504] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb946fbf90da4cb59b723f4b8cd53fdd" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 45315 } } peers { permanent_uuid: "64316452fcad4c9cbe6cfb813692bba0" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44927 } } peers { permanent_uuid: "74916471773d44fdab143f409fa2eaeb" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34961 } }
I20260620 21:45:44.375624 6221 leader_election.cc:288] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [CANDIDATE]: Term 1 election: Requested vote from peers fb946fbf90da4cb59b723f4b8cd53fdd (127.1.155.129:45315), 64316452fcad4c9cbe6cfb813692bba0 (127.1.155.131:44927)
I20260620 21:45:44.376111 5884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d7d283e43f4ca69cc3f952615c2748" candidate_uuid: "74916471773d44fdab143f409fa2eaeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb946fbf90da4cb59b723f4b8cd53fdd"
I20260620 21:45:44.376143 6150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d7d283e43f4ca69cc3f952615c2748" candidate_uuid: "74916471773d44fdab143f409fa2eaeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64316452fcad4c9cbe6cfb813692bba0"
I20260620 21:45:44.376268 5884 raft_consensus.cc:3037] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:44.376271 6150 raft_consensus.cc:3037] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:44.377657 6150 raft_consensus.cc:2454] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74916471773d44fdab143f409fa2eaeb in term 1.
I20260620 21:45:44.377657 5884 raft_consensus.cc:2454] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74916471773d44fdab143f409fa2eaeb in term 1.
I20260620 21:45:44.378072 5951 leader_election.cc:302] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74916471773d44fdab143f409fa2eaeb, fb946fbf90da4cb59b723f4b8cd53fdd; no voters:
I20260620 21:45:44.378356 6221 raft_consensus.cc:2781] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:44.378614 6221 raft_consensus.cc:686] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [term 1 LEADER]: Becoming Leader. State: Replica: 74916471773d44fdab143f409fa2eaeb, State: Running, Role: LEADER
I20260620 21:45:44.379029 6221 consensus_queue.cc:237] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb946fbf90da4cb59b723f4b8cd53fdd" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 45315 } } peers { permanent_uuid: "64316452fcad4c9cbe6cfb813692bba0" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44927 } } peers { permanent_uuid: "74916471773d44fdab143f409fa2eaeb" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34961 } }
I20260620 21:45:44.382223 5739 catalog_manager.cc:5495] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 74916471773d44fdab143f409fa2eaeb (127.1.155.130). New cstate: current_term: 1 leader_uuid: "74916471773d44fdab143f409fa2eaeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb946fbf90da4cb59b723f4b8cd53fdd" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64316452fcad4c9cbe6cfb813692bba0" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74916471773d44fdab143f409fa2eaeb" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34961 } health_report { overall_health: HEALTHY } } }
W20260620 21:45:44.402592 5931 tablet.cc:2369] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:45:44.429047 6212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.425341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.1.155.130:34961)
0620 21:45:44.425495 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:44.425499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:44.425510 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:45:44.425592 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:44.425594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:44.425645 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:44.425762 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:44.426037 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.426648 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.427513 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:44.427733 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:44.427838 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:44.428161 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:44.428647 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:44.428773 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:44.428831 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 21:45:44.428889 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:45:44.435024 5884 raft_consensus.cc:1261] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd [term 1 FOLLOWER]: Refusing update from remote peer 74916471773d44fdab143f409fa2eaeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:44.435024 6150 raft_consensus.cc:1261] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0 [term 1 FOLLOWER]: Refusing update from remote peer 74916471773d44fdab143f409fa2eaeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:44.435813 6223 consensus_queue.cc:1035] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64316452fcad4c9cbe6cfb813692bba0" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:44.436028 6221 consensus_queue.cc:1035] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb946fbf90da4cb59b723f4b8cd53fdd" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:44.445904 6228 mvcc.cc:205] Tried to move back new op lower bound from 7299039004397715456 to 7299039004179095552. Current Snapshot: MvccSnapshot[applied={T|T < 7299039004397715456}]
I20260620 21:45:44.453575 5740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:44.453747 5740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:44.456089 5740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc29f89b568942d4b8dffdcfde1ae133: Sending DeleteTablet for 3 replicas of tablet 26d7d283e43f4ca69cc3f952615c2748
I20260620 21:45:44.457072 5997 tablet_service.cc:1514] Processing DeleteTablet for tablet 26d7d283e43f4ca69cc3f952615c2748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:44 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:32956
I20260620 21:45:44.457093 5864 tablet_service.cc:1514] Processing DeleteTablet for tablet 26d7d283e43f4ca69cc3f952615c2748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:44 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:57182
I20260620 21:45:44.457327 6130 tablet_service.cc:1514] Processing DeleteTablet for tablet 26d7d283e43f4ca69cc3f952615c2748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:44 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:54344
I20260620 21:45:44.457587 6237 tablet_replica.cc:330] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb: stopping tablet replica
I20260620 21:45:44.457767 6238 tablet_replica.cc:330] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd: stopping tablet replica
I20260620 21:45:44.458001 6239 tablet_replica.cc:330] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0: stopping tablet replica
I20260620 21:45:44.458009 6237 raft_consensus.cc:2229] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:45:44.458076 6238 raft_consensus.cc:2229] T 26d7d283e43f4ca69cc3f952615c2748 P fb946fbf90da4cb59b723f4b8cd53fdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:44.458336 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 5802
I20260620 21:45:44.458513 6237 raft_consensus.cc:2258] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:44.460693 6237 ts_tablet_manager.cc:1905] T 26d7d283e43f4ca69cc3f952615c2748 P 74916471773d44fdab143f409fa2eaeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:44.458375 6239 raft_consensus.cc:2229] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:44.462270 6239 raft_consensus.cc:2258] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:44.463876 6239 ts_tablet_manager.cc:1905] T 26d7d283e43f4ca69cc3f952615c2748 P 64316452fcad4c9cbe6cfb813692bba0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 21:45:44.465399 5725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.155.129:45315 (error 108)
I20260620 21:45:44.466195 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 5935
I20260620 21:45:44.466457 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.466157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:45315 (local address 127.0.0.1:53366)
0620 21:45:44.466291 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:44.466361 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 21:45:44.467257 5725 catalog_manager.cc:4584] TS fb946fbf90da4cb59b723f4b8cd53fdd (127.1.155.129:45315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26d7d283e43f4ca69cc3f952615c2748: Network error: Client connection negotiation failed: client connection to 127.1.155.129:45315: connect: Connection refused (error 111)
I20260620 21:45:44.478807 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 6068
I20260620 21:45:44.479058 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.478712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:34961 (local address 127.0.0.1:53342)
0620 21:45:44.478866 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:44.478945 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.130:34961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 21:45:44.485067 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 5709
2026-06-20T21:45:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2505 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T21:45:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:44Z Disabled control of system clock
I20260620 21:45:44.515836 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:42331
--webserver_interface=127.1.155.190
--webserver_port=0
--builtin_ntp_servers=127.1.155.148:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:42331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 21:45:44.625429 6244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:44.625739 6244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:44.625811 6244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:44.629935 6244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:44.630019 6244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:44.630048 6244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:44.630102 6244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:44.630136 6244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:45:44.634889 6244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:42331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:42331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:44.636004 6244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:44.637385 6244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:44.645692 6252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:44.645704 6249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:44.645963 6244 server_base.cc:1034] running on GCE node
W20260620 21:45:44.645704 6250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:44.646539 6244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:44.647596 6244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:44.648761 6244 hybrid_clock.cc:648] HybridClock initialized: now 1781991944648746 us; error 26 us; skew 500 ppm
I20260620 21:45:44.650717 6244 webserver.cc:460] Webserver started at http://127.1.155.190:40123/ using document root <none> and password file <none>
I20260620 21:45:44.651293 6244 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:44.651383 6244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:44.651603 6244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:44.653628 6244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "efe09f3f692340ec905776fe74081aab"
format_stamp: "Formatted at 2026-06-20 21:45:44 on dist-test-slave-5bvn"
server_key: "ced767f5349d121575dfe38dfab72199"
server_key_iv: "14d64b6c300f9dadcb2ff03cdf3c12eb"
server_key_version: "encryptionkey@0"
I20260620 21:45:44.654264 6244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "efe09f3f692340ec905776fe74081aab"
format_stamp: "Formatted at 2026-06-20 21:45:44 on dist-test-slave-5bvn"
server_key: "ced767f5349d121575dfe38dfab72199"
server_key_iv: "14d64b6c300f9dadcb2ff03cdf3c12eb"
server_key_version: "encryptionkey@0"
I20260620 21:45:44.657824 6244 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:45:44.660271 6258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:44.661459 6244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:45:44.661600 6244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "efe09f3f692340ec905776fe74081aab"
format_stamp: "Formatted at 2026-06-20 21:45:44 on dist-test-slave-5bvn"
server_key: "ced767f5349d121575dfe38dfab72199"
server_key_iv: "14d64b6c300f9dadcb2ff03cdf3c12eb"
server_key_version: "encryptionkey@0"
I20260620 21:45:44.661715 6244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:44.666852 6244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:44.667441 6244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:44.667618 6244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:44.675089 6244 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:42331
I20260620 21:45:44.675143 6310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:42331 every 8 connection(s)
I20260620 21:45:44.676025 6244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:44.679247 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:44.681818 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 6244
I20260620 21:45:44.681944 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:44.682228 1646 external_mini_cluster.cc:1428] Setting key e4fd4ddf1eb7383f5ff5c9a7d09d0bb3
I20260620 21:45:44.685307 6311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab: Bootstrap starting.
I20260620 21:45:44.687654 6311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:44.687732 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:44.683698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59688 (local address 127.1.155.190:42331)
0620 21:45:44.684388 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:44.684405 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:44.684439 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:45:44.685051 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:44.685067 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:44.685442 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:44.685765 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:44.686534 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:44.686538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:44.686549 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:44.686558 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:44.686610 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:44.686624 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:44.686771 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:44.686841 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 21:45:44.687062 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":128,"threads_started":1}
I20260620 21:45:44.688395 6311 log.cc:824] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:44.690411 6311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab: No bootstrap required, opened a new log
I20260620 21:45:44.694156 6311 raft_consensus.cc:348] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe09f3f692340ec905776fe74081aab" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42331 } }
I20260620 21:45:44.694389 6311 raft_consensus.cc:374] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:44.694463 6311 raft_consensus.cc:729] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efe09f3f692340ec905776fe74081aab, State: Initialized, Role: FOLLOWER
I20260620 21:45:44.694859 6311 consensus_queue.cc:260] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe09f3f692340ec905776fe74081aab" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42331 } }
I20260620 21:45:44.695003 6311 raft_consensus.cc:388] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:44.695081 6311 raft_consensus.cc:482] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:44.695200 6311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:44.773782 6311 raft_consensus.cc:504] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe09f3f692340ec905776fe74081aab" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42331 } }
I20260620 21:45:44.774441 6311 leader_election.cc:302] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efe09f3f692340ec905776fe74081aab; no voters:
I20260620 21:45:44.774840 6311 leader_election.cc:288] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:44.774991 6316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:44.775243 6316 raft_consensus.cc:686] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [term 1 LEADER]: Becoming Leader. State: Replica: efe09f3f692340ec905776fe74081aab, State: Running, Role: LEADER
I20260620 21:45:44.775622 6316 consensus_queue.cc:237] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe09f3f692340ec905776fe74081aab" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42331 } }
I20260620 21:45:44.776268 6311 sys_catalog.cc:564] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:44.777489 6317 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efe09f3f692340ec905776fe74081aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe09f3f692340ec905776fe74081aab" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42331 } } }
I20260620 21:45:44.777637 6317 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:44.777820 6318 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [sys.catalog]: SysCatalogTable state changed. Reason: New leader efe09f3f692340ec905776fe74081aab. Latest consensus state: current_term: 1 leader_uuid: "efe09f3f692340ec905776fe74081aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe09f3f692340ec905776fe74081aab" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42331 } } }
I20260620 21:45:44.777983 6318 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:44.778244 6325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:44.781164 6325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:44.786984 6325 catalog_manager.cc:1347] Generated new cluster ID: dbaf0bb6550444269f7f0829ce1a96ba
I20260620 21:45:44.787075 6325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:44.804360 6325 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:44.804761 6325 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:44.812664 6325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab: Generated new TSK 0
I20260620 21:45:44.813541 6325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:45:44.831795 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42331
--builtin_ntp_servers=127.1.155.148:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 21:45:44.944433 6335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:44.944689 6335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:44.944728 6335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:44.948784 6335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:44.948839 6335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:44.948930 6335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:45:44.953617 6335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:44.954742 6335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:44.956116 6335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:44.965736 6343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:44.966104 6335 server_base.cc:1034] running on GCE node
W20260620 21:45:44.966363 6340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:44.966378 6341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:44.966904 6335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:44.967653 6335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:44.968820 6335 hybrid_clock.cc:648] HybridClock initialized: now 1781991944968786 us; error 44 us; skew 500 ppm
I20260620 21:45:44.970728 6335 webserver.cc:460] Webserver started at http://127.1.155.129:34011/ using document root <none> and password file <none>
I20260620 21:45:44.971306 6335 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:44.971369 6335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:44.971572 6335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:44.973809 6335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "c76dd8dd7f524608ac6ef4a56484791d"
format_stamp: "Formatted at 2026-06-20 21:45:44 on dist-test-slave-5bvn"
server_key: "f28276cec11812ed5dd3c1ec284da989"
server_key_iv: "e9adba7a08259b14725ab5c725e08fd0"
server_key_version: "encryptionkey@0"
I20260620 21:45:44.974473 6335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "c76dd8dd7f524608ac6ef4a56484791d"
format_stamp: "Formatted at 2026-06-20 21:45:44 on dist-test-slave-5bvn"
server_key: "f28276cec11812ed5dd3c1ec284da989"
server_key_iv: "e9adba7a08259b14725ab5c725e08fd0"
server_key_version: "encryptionkey@0"
I20260620 21:45:44.979252 6335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:45:44.981972 6349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:44.983342 6335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:45:44.983484 6335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "c76dd8dd7f524608ac6ef4a56484791d"
format_stamp: "Formatted at 2026-06-20 21:45:44 on dist-test-slave-5bvn"
server_key: "f28276cec11812ed5dd3c1ec284da989"
server_key_iv: "e9adba7a08259b14725ab5c725e08fd0"
server_key_version: "encryptionkey@0"
I20260620 21:45:44.983606 6335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:44.988435 6335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:44.989086 6335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:44.989257 6335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:44.989791 6335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:44.990742 6335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:44.990809 6335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:44.990872 6335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:44.990917 6335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:44.999950 6335 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:44203
I20260620 21:45:45.000011 6462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:44203 every 8 connection(s)
I20260620 21:45:45.000934 6335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:45:45.007030 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.002889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:45613 (local address 127.1.155.190:42331)
0620 21:45:45.003026 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:45.003029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:45.004043 (+ 1014us) server_negotiation.cc:407] Connection header received
0620 21:45:45.005379 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:45.005382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:45.005419 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:45.005506 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:45.006662 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:45.006665 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:45.006667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:45.006679 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:45.006697 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:45.006705 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:45.006796 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:45.006888 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 21:45:45.006922 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:45:45.007910 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.003256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42331 (local address 127.1.155.129:45613)
0620 21:45:45.003922 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:45.003961 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:45.005122 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:45.005666 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:45.005673 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:45.005764 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:45.006362 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:45:45.006363 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:45.006424 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:45:45.006515 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:45.006780 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:45.006787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:45.006791 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:45:45.006921 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 21:45:45.007113 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260620 21:45:45.008119 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 6335
I20260620 21:45:45.008245 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:45:45.008527 1646 external_mini_cluster.cc:1428] Setting key d8a85ce4eb3238c777f9ebc6026783a3
I20260620 21:45:45.009204 6463 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42331
I20260620 21:45:45.009440 6463 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:45.009919 6463 heartbeater.cc:511] Master 127.1.155.190:42331 requested a full tablet report, sending...
I20260620 21:45:45.010468 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42331
--builtin_ntp_servers=127.1.155.148:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 21:45:45.011548 6273 ts_manager.cc:194] Registered new tserver with Master: c76dd8dd7f524608ac6ef4a56484791d (127.1.155.129:44203)
W20260620 21:45:45.124562 6466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:45.124859 6466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:45.124900 6466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:45.129129 6466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:45.129194 6466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:45.129252 6466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:45:45.134148 6466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:45.135280 6466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:45.136699 6466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:45.146085 6471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:45.146093 6472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:45.146070 6474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:45.146472 6466 server_base.cc:1034] running on GCE node
I20260620 21:45:45.146890 6466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:45.147544 6466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:45.148725 6466 hybrid_clock.cc:648] HybridClock initialized: now 1781991945148694 us; error 45 us; skew 500 ppm
I20260620 21:45:45.150542 6466 webserver.cc:460] Webserver started at http://127.1.155.130:39349/ using document root <none> and password file <none>
I20260620 21:45:45.151129 6466 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:45.151192 6466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:45.151392 6466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:45.153409 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "64f8a6a1b97d44bc8955a7a314883019"
format_stamp: "Formatted at 2026-06-20 21:45:45 on dist-test-slave-5bvn"
server_key: "5fc91d1e93c44d9fd72c5e59b36cb030"
server_key_iv: "a6b47da1586384f118f3b1827e13b4f2"
server_key_version: "encryptionkey@0"
I20260620 21:45:45.154052 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "64f8a6a1b97d44bc8955a7a314883019"
format_stamp: "Formatted at 2026-06-20 21:45:45 on dist-test-slave-5bvn"
server_key: "5fc91d1e93c44d9fd72c5e59b36cb030"
server_key_iv: "a6b47da1586384f118f3b1827e13b4f2"
server_key_version: "encryptionkey@0"
I20260620 21:45:45.158193 6466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:45:45.160791 6480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:45.162206 6466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:45:45.162335 6466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "64f8a6a1b97d44bc8955a7a314883019"
format_stamp: "Formatted at 2026-06-20 21:45:45 on dist-test-slave-5bvn"
server_key: "5fc91d1e93c44d9fd72c5e59b36cb030"
server_key_iv: "a6b47da1586384f118f3b1827e13b4f2"
server_key_version: "encryptionkey@0"
I20260620 21:45:45.162446 6466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:45.167460 6466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:45.168133 6466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:45.168272 6466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:45.168797 6466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:45.169713 6466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:45.169775 6466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:45.169827 6466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:45.169888 6466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:45.178472 6466 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:36857
I20260620 21:45:45.178563 6593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:36857 every 8 connection(s)
I20260620 21:45:45.179438 6466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:45:45.186054 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.181509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:43683 (local address 127.1.155.190:42331)
0620 21:45:45.181649 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:45.181653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:45.182995 (+ 1342us) server_negotiation.cc:407] Connection header received
0620 21:45:45.184342 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:45.184345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:45.184390 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:45.184496 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:45.185551 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:45.185555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:45.185557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:45.185571 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:45.185599 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:45.185607 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:45.185704 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:45.185877 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 21:45:45.185922 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:45:45.186975 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.182125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42331 (local address 127.1.155.130:43683)
0620 21:45:45.182845 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:45.182886 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:45.184138 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:45.184654 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:45.184661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:45.184760 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:45.185306 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:45:45.185307 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:45.185343 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:45:45.185406 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:45.185689 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:45.185697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:45.185702 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:45:45.185910 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 21:45:45.186137 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":141,"threads_started":1}
I20260620 21:45:45.187798 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 6466
I20260620 21:45:45.187928 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:45:45.188215 1646 external_mini_cluster.cc:1428] Setting key 75e33734b9ee67b5fd06747399469a1a
I20260620 21:45:45.188227 6594 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42331
I20260620 21:45:45.188431 6594 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:45.188820 6594 heartbeater.cc:511] Master 127.1.155.190:42331 requested a full tablet report, sending...
I20260620 21:45:45.189909 6273 ts_manager.cc:194] Registered new tserver with Master: 64f8a6a1b97d44bc8955a7a314883019 (127.1.155.130:36857)
I20260620 21:45:45.190371 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42331
--builtin_ntp_servers=127.1.155.148:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 21:45:45.305752 6597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:45.306160 6597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:45.306238 6597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:45.310434 6597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:45.310493 6597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:45.310554 6597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:45:45.315547 6597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:45.316713 6597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:45.318126 6597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:45.327593 6603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:45.327605 6605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:45.327617 6602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:45.327654 6597 server_base.cc:1034] running on GCE node
I20260620 21:45:45.328212 6597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:45.328855 6597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:45.330034 6597 hybrid_clock.cc:648] HybridClock initialized: now 1781991945330012 us; error 40 us; skew 500 ppm
I20260620 21:45:45.331830 6597 webserver.cc:460] Webserver started at http://127.1.155.131:36703/ using document root <none> and password file <none>
I20260620 21:45:45.332407 6597 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:45.332504 6597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:45.332739 6597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:45.334846 6597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "573afbe2dee94af9bc4201c07236a914"
format_stamp: "Formatted at 2026-06-20 21:45:45 on dist-test-slave-5bvn"
server_key: "5ea92116e9bfa97f35cf518177053364"
server_key_iv: "2547efbdd59578f0091e8f77117d6f9d"
server_key_version: "encryptionkey@0"
I20260620 21:45:45.335402 6597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "573afbe2dee94af9bc4201c07236a914"
format_stamp: "Formatted at 2026-06-20 21:45:45 on dist-test-slave-5bvn"
server_key: "5ea92116e9bfa97f35cf518177053364"
server_key_iv: "2547efbdd59578f0091e8f77117d6f9d"
server_key_version: "encryptionkey@0"
I20260620 21:45:45.339156 6597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:45:45.341516 6611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:45.342934 6597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:45:45.343062 6597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "573afbe2dee94af9bc4201c07236a914"
format_stamp: "Formatted at 2026-06-20 21:45:45 on dist-test-slave-5bvn"
server_key: "5ea92116e9bfa97f35cf518177053364"
server_key_iv: "2547efbdd59578f0091e8f77117d6f9d"
server_key_version: "encryptionkey@0"
I20260620 21:45:45.343174 6597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:45.348388 6597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:45.348987 6597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:45.349157 6597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:45.349664 6597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:45.350543 6597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:45.350589 6597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:45.350649 6597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:45.350685 6597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:45.359222 6597 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:39873
I20260620 21:45:45.359248 6724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:39873 every 8 connection(s)
I20260620 21:45:45.360096 6597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:45:45.366426 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.361891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:44825 (local address 127.1.155.190:42331)
0620 21:45:45.362114 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:45.362120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:45.363137 (+ 1017us) server_negotiation.cc:407] Connection header received
0620 21:45:45.364535 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:45.364539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:45.364584 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:45.364701 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:45.365897 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:45.365903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:45.365905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:45.365919 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:45.365945 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:45.365968 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:45.366104 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:45.366217 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 21:45:45.366261 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260620 21:45:45.366590 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 6597
I20260620 21:45:45.366719 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:45:45.366997 1646 external_mini_cluster.cc:1428] Setting key 74830b3cc39583551fe57bab5d2f194e
I20260620 21:45:45.367372 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.362325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42331 (local address 127.1.155.131:44825)
0620 21:45:45.362985 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:45.363027 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:45.364273 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:45.364859 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:45.364865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:45.364951 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:45.365552 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:45:45.365554 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:45.365592 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:45:45.365673 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:45.366069 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:45.366076 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:45.366080 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:45:45.366246 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 21:45:45.366477 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":116,"threads_started":1}
I20260620 21:45:45.368474 6725 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42331
I20260620 21:45:45.368700 6725 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:45.369136 6725 heartbeater.cc:511] Master 127.1.155.190:42331 requested a full tablet report, sending...
I20260620 21:45:45.370119 6273 ts_manager.cc:194] Registered new tserver with Master: 573afbe2dee94af9bc4201c07236a914 (127.1.155.131:39873)
I20260620 21:45:45.380576 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:45:45.384495 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.382719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59698 (local address 127.1.155.190:42331)
0620 21:45:45.382870 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:45.382876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:45.383120 (+ 244us) server_negotiation.cc:407] Connection header received
0620 21:45:45.383209 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:45.383211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:45.383245 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:45.383311 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:45.384120 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:45.384122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:45.384124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:45.384134 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:45.384150 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:45.384157 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:45.384224 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:45.384323 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 21:45:45.384354 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:45:45.387362 6273 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:45:45.390437 6273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:45.402422 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.399145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.1.155.129:44203)
0620 21:45:45.400277 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:45.400292 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:45.400319 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:45:45.400361 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:45.400365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:45.400496 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:45.400649 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:45.401809 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:45.401817 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:45.401827 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:45.401866 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:45.401910 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:45.401935 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:45.402057 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:45.402193 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 21:45:45.402264 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":147,"threads_started":1}
I20260620 21:45:45.402441 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.399752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.1.155.130:36857)
0620 21:45:45.400174 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:45.400189 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:45.400220 (+ 31us) server_negotiation.cc:407] Connection header received
0620 21:45:45.400280 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:45.400286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:45.400497 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:45.401012 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:45.401823 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:45.401830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:45.401843 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:45.401876 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:45.401925 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:45.401972 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:45.402082 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:45.402209 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 21:45:45.402278 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":107,"threads_started":1}
I20260620 21:45:45.402416 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.398892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:44203 (local address 127.0.0.1:34086)
0620 21:45:45.399311 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:45.399364 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:45.399464 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:45.400643 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:45.400648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:45.400722 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:45.401419 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:45:45.401423 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:45.401448 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:45:45.401524 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:45.402030 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:45.402038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:45.402052 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 21:45:45.402180 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 21:45:45.402250 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":107,"threads_started":1}
I20260620 21:45:45.402422 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.399343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:36857 (local address 127.0.0.1:43408)
0620 21:45:45.399693 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:45.399706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:45.399773 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:45.400670 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:45.400673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:45.400723 (+ 50us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:45.401649 (+ 926us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:45:45.401651 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:45.401660 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:45:45.401680 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:45.402070 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:45.402074 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:45.402078 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:45:45.402198 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 21:45:45.402261 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":3,"thread_start_us":100,"threads_started":1}
I20260620 21:45:45.403601 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.399629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:39873 (local address 127.0.0.1:34480)
0620 21:45:45.400008 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:45.400020 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:45.400088 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:45.401540 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:45.401543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:45.401564 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:45.402888 (+ 1324us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:45:45.402890 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:45.402901 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:45:45.402929 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:45.403324 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:45.403327 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:45.403331 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:45:45.403416 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 21:45:45.403464 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260620 21:45:45.403693 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.400055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.1.155.131:39873)
0620 21:45:45.400476 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:45.400487 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:45.400509 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:45:45.400566 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:45.400574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:45.400723 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:45.400862 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:45.403081 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:45.403087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:45.403097 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:45.403110 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:45.403151 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:45.403182 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:45.403279 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:45.403452 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 21:45:45.403516 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":150,"threads_started":1}
I20260620 21:45:45.405464 6397 tablet_service.cc:1467] Processing CreateTablet for tablet 4a960e301aa14f83bea77df4297ec78f (DEFAULT_TABLE table=test-table [id=b770091b958a422baa0642b0b5999e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:45.405567 6528 tablet_service.cc:1467] Processing CreateTablet for tablet 4a960e301aa14f83bea77df4297ec78f (DEFAULT_TABLE table=test-table [id=b770091b958a422baa0642b0b5999e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:45.406497 6397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a960e301aa14f83bea77df4297ec78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:45.406236 6659 tablet_service.cc:1467] Processing CreateTablet for tablet 4a960e301aa14f83bea77df4297ec78f (DEFAULT_TABLE table=test-table [id=b770091b958a422baa0642b0b5999e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:45.406626 6528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a960e301aa14f83bea77df4297ec78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:45.407120 6659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a960e301aa14f83bea77df4297ec78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:45.412535 6742 tablet_bootstrap.cc:492] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d: Bootstrap starting.
I20260620 21:45:45.413968 6743 tablet_bootstrap.cc:492] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019: Bootstrap starting.
I20260620 21:45:45.414345 6744 tablet_bootstrap.cc:492] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914: Bootstrap starting.
I20260620 21:45:45.414958 6742 tablet_bootstrap.cc:654] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:45.415859 6742 log.cc:824] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:45.416438 6744 tablet_bootstrap.cc:654] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:45.417249 6744 log.cc:824] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:45.417884 6742 tablet_bootstrap.cc:492] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d: No bootstrap required, opened a new log
I20260620 21:45:45.418081 6742 ts_tablet_manager.cc:1397] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:45:45.418367 6743 tablet_bootstrap.cc:654] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:45.419138 6743 log.cc:824] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:45.419183 6744 tablet_bootstrap.cc:492] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914: No bootstrap required, opened a new log
I20260620 21:45:45.419343 6744 ts_tablet_manager.cc:1397] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 21:45:45.420982 6743 tablet_bootstrap.cc:492] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019: No bootstrap required, opened a new log
I20260620 21:45:45.421232 6743 ts_tablet_manager.cc:1397] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 21:45:45.422103 6742 raft_consensus.cc:348] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76dd8dd7f524608ac6ef4a56484791d" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44203 } } peers { permanent_uuid: "573afbe2dee94af9bc4201c07236a914" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39873 } } peers { permanent_uuid: "64f8a6a1b97d44bc8955a7a314883019" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36857 } }
I20260620 21:45:45.422329 6742 raft_consensus.cc:374] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:45.422399 6742 raft_consensus.cc:729] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c76dd8dd7f524608ac6ef4a56484791d, State: Initialized, Role: FOLLOWER
I20260620 21:45:45.422832 6742 consensus_queue.cc:260] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76dd8dd7f524608ac6ef4a56484791d" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44203 } } peers { permanent_uuid: "573afbe2dee94af9bc4201c07236a914" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39873 } } peers { permanent_uuid: "64f8a6a1b97d44bc8955a7a314883019" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36857 } }
I20260620 21:45:45.423453 6463 heartbeater.cc:503] Master 127.1.155.190:42331 was elected leader, sending a full tablet report...
I20260620 21:45:45.423753 6742 ts_tablet_manager.cc:1428] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:45:45.425899 6743 raft_consensus.cc:348] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76dd8dd7f524608ac6ef4a56484791d" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44203 } } peers { permanent_uuid: "573afbe2dee94af9bc4201c07236a914" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39873 } } peers { permanent_uuid: "64f8a6a1b97d44bc8955a7a314883019" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36857 } }
I20260620 21:45:45.426182 6743 raft_consensus.cc:374] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:45.426051 6744 raft_consensus.cc:348] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76dd8dd7f524608ac6ef4a56484791d" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44203 } } peers { permanent_uuid: "573afbe2dee94af9bc4201c07236a914" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39873 } } peers { permanent_uuid: "64f8a6a1b97d44bc8955a7a314883019" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36857 } }
I20260620 21:45:45.426294 6743 raft_consensus.cc:729] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f8a6a1b97d44bc8955a7a314883019, State: Initialized, Role: FOLLOWER
I20260620 21:45:45.426313 6744 raft_consensus.cc:374] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:45.426362 6744 raft_consensus.cc:729] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573afbe2dee94af9bc4201c07236a914, State: Initialized, Role: FOLLOWER
I20260620 21:45:45.426795 6743 consensus_queue.cc:260] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76dd8dd7f524608ac6ef4a56484791d" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44203 } } peers { permanent_uuid: "573afbe2dee94af9bc4201c07236a914" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39873 } } peers { permanent_uuid: "64f8a6a1b97d44bc8955a7a314883019" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36857 } }
I20260620 21:45:45.426795 6744 consensus_queue.cc:260] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76dd8dd7f524608ac6ef4a56484791d" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44203 } } peers { permanent_uuid: "573afbe2dee94af9bc4201c07236a914" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39873 } } peers { permanent_uuid: "64f8a6a1b97d44bc8955a7a314883019" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36857 } }
I20260620 21:45:45.427665 6594 heartbeater.cc:503] Master 127.1.155.190:42331 was elected leader, sending a full tablet report...
I20260620 21:45:45.427754 6743 ts_tablet_manager.cc:1428] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:45:45.427754 6744 ts_tablet_manager.cc:1428] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260620 21:45:45.428149 6725 heartbeater.cc:503] Master 127.1.155.190:42331 was elected leader, sending a full tablet report...
W20260620 21:45:45.430303 6595 tablet.cc:2369] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:45:45.502175 6464 tablet.cc:2369] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:45:45.611073 6726 tablet.cc:2369] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:45:45.682627 6748 raft_consensus.cc:482] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:45.682838 6748 raft_consensus.cc:504] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76dd8dd7f524608ac6ef4a56484791d" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44203 } } peers { permanent_uuid: "573afbe2dee94af9bc4201c07236a914" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39873 } } peers { permanent_uuid: "64f8a6a1b97d44bc8955a7a314883019" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36857 } }
I20260620 21:45:45.684209 6748 leader_election.cc:288] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 573afbe2dee94af9bc4201c07236a914 (127.1.155.131:39873), 64f8a6a1b97d44bc8955a7a314883019 (127.1.155.130:36857)
I20260620 21:45:45.686942 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.684828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:51105 (local address 127.1.155.131:39873)
0620 21:45:45.684947 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:45.684952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:45.685079 (+ 127us) server_negotiation.cc:407] Connection header received
0620 21:45:45.685241 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:45.685245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:45.685280 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:45.685329 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:45.686412 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:45.686416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:45.686419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:45.686433 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:45.686455 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:45.686465 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:45.686562 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:45.686754 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 21:45:45.686797 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 21:45:45.687129 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.685031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:42355 (local address 127.1.155.130:36857)
0620 21:45:45.685168 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:45.685174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:45.685441 (+ 267us) server_negotiation.cc:407] Connection header received
0620 21:45:45.685618 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:45.685625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:45.685665 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:45.685718 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:45.686605 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:45.686612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:45.686618 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:45.686633 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:45.686655 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:45.686667 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:45.686736 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:45.686900 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 21:45:45.686957 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 21:45:45.687301 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.684451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:39873 (local address 127.1.155.129:51105)
0620 21:45:45.684998 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:45.685012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:45.685139 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:45.685445 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:45.685448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:45.685462 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:45.686237 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:45:45.686241 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:45.686255 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:45:45.686289 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:45.686603 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:45.686606 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:45.686610 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:45:45.687137 (+ 527us) client_negotiation.cc:240] Negotiation successful
0620 21:45:45.687174 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":94,"threads_started":1}
I20260620 21:45:45.687975 6679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a960e301aa14f83bea77df4297ec78f" candidate_uuid: "c76dd8dd7f524608ac6ef4a56484791d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573afbe2dee94af9bc4201c07236a914" is_pre_election: true
I20260620 21:45:45.688344 6679 raft_consensus.cc:2454] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c76dd8dd7f524608ac6ef4a56484791d in term 0.
I20260620 21:45:45.688856 6350 leader_election.cc:302] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573afbe2dee94af9bc4201c07236a914, c76dd8dd7f524608ac6ef4a56484791d; no voters:
I20260620 21:45:45.689108 6748 raft_consensus.cc:2781] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:45.689153 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.684677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:36857 (local address 127.1.155.129:42355)
0620 21:45:45.685359 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:45.685375 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:45.685500 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:45.685859 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:45.685865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:45.685884 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:45.686432 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:45:45.686436 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:45.686447 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:45:45.686470 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:45.686802 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:45.686807 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:45.686813 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:45:45.687458 (+ 645us) client_negotiation.cc:240] Negotiation successful
0620 21:45:45.687494 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":74,"threads_started":1}
I20260620 21:45:45.689186 6748 raft_consensus.cc:482] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:45.689236 6748 raft_consensus.cc:3037] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:45.689728 6548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a960e301aa14f83bea77df4297ec78f" candidate_uuid: "c76dd8dd7f524608ac6ef4a56484791d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f8a6a1b97d44bc8955a7a314883019" is_pre_election: true
I20260620 21:45:45.690012 6548 raft_consensus.cc:2454] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c76dd8dd7f524608ac6ef4a56484791d in term 0.
I20260620 21:45:45.784902 6748 raft_consensus.cc:504] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76dd8dd7f524608ac6ef4a56484791d" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44203 } } peers { permanent_uuid: "573afbe2dee94af9bc4201c07236a914" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39873 } } peers { permanent_uuid: "64f8a6a1b97d44bc8955a7a314883019" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36857 } }
I20260620 21:45:45.785434 6748 leader_election.cc:288] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [CANDIDATE]: Term 1 election: Requested vote from peers 573afbe2dee94af9bc4201c07236a914 (127.1.155.131:39873), 64f8a6a1b97d44bc8955a7a314883019 (127.1.155.130:36857)
I20260620 21:45:45.785970 6548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a960e301aa14f83bea77df4297ec78f" candidate_uuid: "c76dd8dd7f524608ac6ef4a56484791d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f8a6a1b97d44bc8955a7a314883019"
I20260620 21:45:45.785957 6679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a960e301aa14f83bea77df4297ec78f" candidate_uuid: "c76dd8dd7f524608ac6ef4a56484791d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573afbe2dee94af9bc4201c07236a914"
I20260620 21:45:45.786135 6548 raft_consensus.cc:3037] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:45.786128 6679 raft_consensus.cc:3037] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:45.787379 6548 raft_consensus.cc:2454] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c76dd8dd7f524608ac6ef4a56484791d in term 1.
I20260620 21:45:45.787804 6679 raft_consensus.cc:2454] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c76dd8dd7f524608ac6ef4a56484791d in term 1.
I20260620 21:45:45.787808 6353 leader_election.cc:302] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f8a6a1b97d44bc8955a7a314883019, c76dd8dd7f524608ac6ef4a56484791d; no voters:
I20260620 21:45:45.788075 6748 raft_consensus.cc:2781] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:45.788344 6748 raft_consensus.cc:686] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [term 1 LEADER]: Becoming Leader. State: Replica: c76dd8dd7f524608ac6ef4a56484791d, State: Running, Role: LEADER
I20260620 21:45:45.788727 6748 consensus_queue.cc:237] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76dd8dd7f524608ac6ef4a56484791d" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44203 } } peers { permanent_uuid: "573afbe2dee94af9bc4201c07236a914" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39873 } } peers { permanent_uuid: "64f8a6a1b97d44bc8955a7a314883019" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36857 } }
I20260620 21:45:45.791918 6273 catalog_manager.cc:5495] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d reported cstate change: term changed from 0 to 1, leader changed from <none> to c76dd8dd7f524608ac6ef4a56484791d (127.1.155.129). New cstate: current_term: 1 leader_uuid: "c76dd8dd7f524608ac6ef4a56484791d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76dd8dd7f524608ac6ef4a56484791d" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 44203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "573afbe2dee94af9bc4201c07236a914" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64f8a6a1b97d44bc8955a7a314883019" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36857 } health_report { overall_health: UNKNOWN } } }
I20260620 21:45:45.853590 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.852073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.1.155.129:44203)
0620 21:45:45.852187 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:45.852190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:45.852203 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:45:45.852308 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:45.852311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:45.852377 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:45.852457 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:45.853176 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:45.853179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:45.853181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:45.853192 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:45.853210 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:45.853217 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:45.853287 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:45.853428 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 21:45:45.853474 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 21:45:45.859320 6679 raft_consensus.cc:1261] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914 [term 1 FOLLOWER]: Refusing update from remote peer c76dd8dd7f524608ac6ef4a56484791d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:45.859320 6548 raft_consensus.cc:1261] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019 [term 1 FOLLOWER]: Refusing update from remote peer c76dd8dd7f524608ac6ef4a56484791d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:45.859948 6748 consensus_queue.cc:1035] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [LEADER]: Connected to new peer: Peer: permanent_uuid: "573afbe2dee94af9bc4201c07236a914" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:45.860142 6753 consensus_queue.cc:1035] T 4a960e301aa14f83bea77df4297ec78f P c76dd8dd7f524608ac6ef4a56484791d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f8a6a1b97d44bc8955a7a314883019" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:45.866839 6756 mvcc.cc:205] Tried to move back new op lower bound from 7299039010232872960 to 7299039009953267712. Current Snapshot: MvccSnapshot[applied={T|T < 7299039010232872960}]
I20260620 21:45:45.867659 6758 mvcc.cc:205] Tried to move back new op lower bound from 7299039010232872960 to 7299039009953267712. Current Snapshot: MvccSnapshot[applied={T|T < 7299039010232872960}]
I20260620 21:45:45.868199 6757 mvcc.cc:205] Tried to move back new op lower bound from 7299039010232872960 to 7299039009953267712. Current Snapshot: MvccSnapshot[applied={T|T < 7299039010232872960}]
I20260620 21:45:45.877789 6269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:45.878010 6269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:45.880730 6269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efe09f3f692340ec905776fe74081aab: Sending DeleteTablet for 3 replicas of tablet 4a960e301aa14f83bea77df4297ec78f
I20260620 21:45:45.881664 6528 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a960e301aa14f83bea77df4297ec78f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:45 UTC) from {username='slave'} at 127.0.0.1:43408
I20260620 21:45:45.882175 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 6335
I20260620 21:45:45.882246 6659 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a960e301aa14f83bea77df4297ec78f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:45 UTC) from {username='slave'} at 127.0.0.1:34480
I20260620 21:45:45.883082 6767 tablet_replica.cc:330] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914: stopping tablet replica
I20260620 21:45:45.883672 6767 raft_consensus.cc:2229] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:45.883853 6766 tablet_replica.cc:330] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019: stopping tablet replica
I20260620 21:45:45.884027 6767 raft_consensus.cc:2258] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:45.884135 6766 raft_consensus.cc:2229] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:45.885325 6767 ts_tablet_manager.cc:1905] T 4a960e301aa14f83bea77df4297ec78f P 573afbe2dee94af9bc4201c07236a914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:45.889006 6766 raft_consensus.cc:2258] T 4a960e301aa14f83bea77df4297ec78f P 64f8a6a1b97d44bc8955a7a314883019 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 21:45:45.890092 6259 connection.cc:535] client connection to 127.1.155.129:44203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 21:45:45.890254 6259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 21:45:45.890363 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 6466
I20260620 21:45:45.891717 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.891351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:44203 (local address 127.0.0.1:34100)
0620 21:45:45.891516 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:45.891609 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 21:45:45.891894 6259 catalog_manager.cc:4584] TS c76dd8dd7f524608ac6ef4a56484791d (127.1.155.129:44203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a960e301aa14f83bea77df4297ec78f: Network error: Client connection negotiation failed: client connection to 127.1.155.129:44203: connect: Connection refused (error 111)
I20260620 21:45:45.896319 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 6597
I20260620 21:45:45.898195 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.897073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:36857 (local address 127.0.0.1:43418)
0620 21:45:45.898073 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:45.898121 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.130:36857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":917}
I20260620 21:45:45.942440 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 6244
I20260620 21:45:45.942786 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:45.942426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:36857 (local address 127.0.0.1:43428)
0620 21:45:45.942585 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:45.942663 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.130:36857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
2026-06-20T21:45:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1455 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 21:45:45.963928 1646 mini_oidc.cc:152] Starting JWKS server
I20260620 21:45:45.965521 1646 webserver.cc:460] Webserver started at https://127.0.0.1:33647/ using document root <none> and password file <none>
I20260620 21:45:45.965698 1646 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 21:45:45.966058 1646 webserver.cc:460] Webserver started at http://127.0.0.1:36821/ using document root <none> and password file <none>
2026-06-20T21:45:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:45Z Disabled control of system clock
I20260620 21:45:45.976763 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:39861
--webserver_interface=127.1.155.190
--webserver_port=0
--builtin_ntp_servers=127.1.155.148:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:39861
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33647/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/testchainca.cert with env {}
W20260620 21:45:46.090355 6774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:46.090643 6774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:46.090684 6774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:46.095178 6774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:46.095254 6774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:46.095271 6774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:46.095289 6774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:46.095304 6774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:45:46.095321 6774 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:45:46.095335 6774 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33647/jwks.json
I20260620 21:45:46.100843 6774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:39861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:39861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33647/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:46.101986 6774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:46.102948 6774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:46.111287 6779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:46.111308 6780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:46.111296 6782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:46.111675 6774 server_base.cc:1034] running on GCE node
I20260620 21:45:46.112282 6774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:46.113368 6774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:46.114565 6774 hybrid_clock.cc:648] HybridClock initialized: now 1781991946114530 us; error 42 us; skew 500 ppm
I20260620 21:45:46.116420 6774 webserver.cc:460] Webserver started at http://127.1.155.190:38947/ using document root <none> and password file <none>
I20260620 21:45:46.117002 6774 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:46.117070 6774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:46.117296 6774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:46.119366 6774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "22c88c732b7049cab3162cfc1a7bb71f"
format_stamp: "Formatted at 2026-06-20 21:45:46 on dist-test-slave-5bvn"
server_key: "04b6b1b4d7f5c6eb3557b6aa078b95b6"
server_key_iv: "51396dd720d5aeeeed479b69b16bf14f"
server_key_version: "encryptionkey@0"
I20260620 21:45:46.119949 6774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "22c88c732b7049cab3162cfc1a7bb71f"
format_stamp: "Formatted at 2026-06-20 21:45:46 on dist-test-slave-5bvn"
server_key: "04b6b1b4d7f5c6eb3557b6aa078b95b6"
server_key_iv: "51396dd720d5aeeeed479b69b16bf14f"
server_key_version: "encryptionkey@0"
I20260620 21:45:46.123623 6774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 21:45:46.126030 6788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:46.127213 6774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:45:46.127348 6774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "22c88c732b7049cab3162cfc1a7bb71f"
format_stamp: "Formatted at 2026-06-20 21:45:46 on dist-test-slave-5bvn"
server_key: "04b6b1b4d7f5c6eb3557b6aa078b95b6"
server_key_iv: "51396dd720d5aeeeed479b69b16bf14f"
server_key_version: "encryptionkey@0"
I20260620 21:45:46.127458 6774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:46.154289 6774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:46.155005 6774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:46.155200 6774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:46.162087 6774 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:39861
I20260620 21:45:46.162129 6840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:39861 every 8 connection(s)
I20260620 21:45:46.163067 6774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:46.166282 6841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:46.171617 6841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f: Bootstrap starting.
I20260620 21:45:46.173125 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 6774
I20260620 21:45:46.173274 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:46.173549 1646 external_mini_cluster.cc:1428] Setting key 2e9c9b9efddfecc11f7d9c802da1bf9c
I20260620 21:45:46.174151 6841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:46.175041 6841 log.cc:824] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:46.177124 6841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f: No bootstrap required, opened a new log
I20260620 21:45:46.181360 6841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c88c732b7049cab3162cfc1a7bb71f" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39861 } }
I20260620 21:45:46.181625 6841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:46.181682 6841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c88c732b7049cab3162cfc1a7bb71f, State: Initialized, Role: FOLLOWER
I20260620 21:45:46.182233 6841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c88c732b7049cab3162cfc1a7bb71f" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39861 } }
I20260620 21:45:46.182381 6841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:46.182471 6841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:46.182566 6841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:46.183156 6844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:46.174685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.1.155.190:39861)
0620 21:45:46.175344 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:46.175370 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:46.175430 (+ 60us) server_negotiation.cc:407] Connection header received
0620 21:45:46.176226 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:46.176245 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:46.176582 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:46.176868 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:46.177245 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.178284 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.179529 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.179999 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.180133 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:46.181075 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:46.181080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:46.181093 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:46.181496 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:46.181550 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:46.181570 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:46.181726 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:46.182000 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 21:45:46.182266 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":142,"threads_started":1}
I20260620 21:45:46.183531 6841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c88c732b7049cab3162cfc1a7bb71f" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39861 } }
I20260620 21:45:46.183854 6841 leader_election.cc:302] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22c88c732b7049cab3162cfc1a7bb71f; no voters:
I20260620 21:45:46.184183 6841 leader_election.cc:288] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:46.184227 6846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:46.184404 6846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [term 1 LEADER]: Becoming Leader. State: Replica: 22c88c732b7049cab3162cfc1a7bb71f, State: Running, Role: LEADER
I20260620 21:45:46.184743 6846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c88c732b7049cab3162cfc1a7bb71f" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39861 } }
I20260620 21:45:46.185395 6841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:46.186146 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:45:46.186298 6848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22c88c732b7049cab3162cfc1a7bb71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c88c732b7049cab3162cfc1a7bb71f" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39861 } } }
I20260620 21:45:46.186427 6848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:46.186924 6854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:46.187162 6847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22c88c732b7049cab3162cfc1a7bb71f. Latest consensus state: current_term: 1 leader_uuid: "22c88c732b7049cab3162cfc1a7bb71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c88c732b7049cab3162cfc1a7bb71f" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39861 } } }
I20260620 21:45:46.188066 6847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:46.190145 6854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:46.195628 6854 catalog_manager.cc:1347] Generated new cluster ID: 554bf119bde8496d903cfba0c1e444de
I20260620 21:45:46.195714 6854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:46.221453 6854 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:46.222760 6854 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:46.237787 6854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22c88c732b7049cab3162cfc1a7bb71f: Generated new TSK 0
I20260620 21:45:46.238685 6854 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 21:45:46.264552 6871 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 21:45:46.264674 6871 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 21:45:46.265287 6844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:46.248258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.1.155.190:39861)
0620 21:45:46.248395 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:46.248400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:46.248582 (+ 182us) server_negotiation.cc:407] Connection header received
0620 21:45:46.248768 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:46.248771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:46.248825 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:46.248942 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:45:46.249176 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.250033 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.250861 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.251215 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.251317 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:46.251627 (+ 310us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:45:46.264847 (+ 13220us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 21:45:46.265135 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60392: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 21:45:46.265369 6844 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60392: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 21:45:46.266475 6870 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:45:46.248137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39861 (local address 127.0.0.1:60392)
0620 21:45:46.248495 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:46.248508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:46.248618 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:46.248951 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:46.248953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:46.248958 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:45:46.249078 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:46.249083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.250131 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.250133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:46.250755 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:46.250762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.251330 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.251334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:46.251403 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:46.251444 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:45:46.265070 (+ 13626us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 21:45:46.265262 (+ 192us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.155.190:39861: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":111,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.155.190:39861: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 21:45:46.267365 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 6774
2026-06-20T21:45:46Z chronyd exiting
I20260620 21:45:46.305023 1646 test_util.cc:175] -----------------------------------------------
I20260620 21:45:46.305152 1646 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781991920067886-1646-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (342 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 21:45:46.306340 1646 mini_oidc.cc:152] Starting JWKS server
I20260620 21:45:46.307235 1646 webserver.cc:460] Webserver started at https://127.0.0.1:45729/ using document root <none> and password file <none>
I20260620 21:45:46.307454 1646 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 21:45:46.307766 1646 webserver.cc:460] Webserver started at http://127.0.0.1:42365/ using document root <none> and password file <none>
2026-06-20T21:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:46Z Disabled control of system clock
I20260620 21:45:46.316709 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:43343
--webserver_interface=127.1.155.190
--webserver_port=0
--builtin_ntp_servers=127.1.155.148:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:43343
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45729/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/testchainca.cert with env {}
W20260620 21:45:46.426359 6877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:46.426625 6877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:46.426664 6877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:46.430598 6877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:46.430650 6877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:46.430666 6877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:46.430683 6877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:46.430699 6877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:45:46.430716 6877 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:45:46.430729 6877 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45729/jwks.json
I20260620 21:45:46.436224 6877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:43343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:43343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45729/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:46.437224 6877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:46.438170 6877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:46.446614 6883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:46.446593 6882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:46.446621 6885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:46.446911 6877 server_base.cc:1034] running on GCE node
I20260620 21:45:46.447515 6877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:46.448586 6877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:46.449741 6877 hybrid_clock.cc:648] HybridClock initialized: now 1781991946449723 us; error 32 us; skew 500 ppm
I20260620 21:45:46.451624 6877 webserver.cc:460] Webserver started at http://127.1.155.190:40131/ using document root <none> and password file <none>
I20260620 21:45:46.452139 6877 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:46.452196 6877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:46.452349 6877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:46.454331 6877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "a362ed19dc29430fa1932aea928d2779"
format_stamp: "Formatted at 2026-06-20 21:45:46 on dist-test-slave-5bvn"
server_key: "f0de8aa4674111e2eb51cb988dc9ab5f"
server_key_iv: "b55e41226b80192f14f320b5fbf0662c"
server_key_version: "encryptionkey@0"
I20260620 21:45:46.454828 6877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "a362ed19dc29430fa1932aea928d2779"
format_stamp: "Formatted at 2026-06-20 21:45:46 on dist-test-slave-5bvn"
server_key: "f0de8aa4674111e2eb51cb988dc9ab5f"
server_key_iv: "b55e41226b80192f14f320b5fbf0662c"
server_key_version: "encryptionkey@0"
I20260620 21:45:46.458277 6877 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:45:46.460546 6891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:46.461737 6877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:45:46.461890 6877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "a362ed19dc29430fa1932aea928d2779"
format_stamp: "Formatted at 2026-06-20 21:45:46 on dist-test-slave-5bvn"
server_key: "f0de8aa4674111e2eb51cb988dc9ab5f"
server_key_iv: "b55e41226b80192f14f320b5fbf0662c"
server_key_version: "encryptionkey@0"
I20260620 21:45:46.462001 6877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:46.482383 6877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:46.483095 6877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:46.483294 6877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:46.490069 6877 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:43343
I20260620 21:45:46.490080 6943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:43343 every 8 connection(s)
I20260620 21:45:46.491000 6877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:46.492378 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 6877
I20260620 21:45:46.492499 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:46.492813 1646 external_mini_cluster.cc:1428] Setting key daf4a08e4d6b3bc8c17be1b2a7e38175
I20260620 21:45:46.494591 6944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:46.500190 6944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779: Bootstrap starting.
I20260620 21:45:46.501768 6947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:46.494224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36224 (local address 127.1.155.190:43343)
0620 21:45:46.494885 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:46.494900 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:46.494938 (+ 38us) server_negotiation.cc:407] Connection header received
0620 21:45:46.495590 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:46.495605 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:46.495974 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:46.496298 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:46.496700 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.497655 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.498624 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.499086 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.499255 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:46.500109 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:46.500113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:46.500126 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:46.500548 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:46.500601 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:46.500621 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:46.500726 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:46.500865 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 21:45:46.501099 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":133,"threads_started":1}
I20260620 21:45:46.502561 6944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:46.503279 6944 log.cc:824] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:46.504576 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:45:46.505419 6944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779: No bootstrap required, opened a new log
I20260620 21:45:46.509090 6944 raft_consensus.cc:348] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a362ed19dc29430fa1932aea928d2779" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43343 } }
I20260620 21:45:46.509320 6944 raft_consensus.cc:374] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:46.509397 6944 raft_consensus.cc:729] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a362ed19dc29430fa1932aea928d2779, State: Initialized, Role: FOLLOWER
I20260620 21:45:46.509872 6944 consensus_queue.cc:260] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a362ed19dc29430fa1932aea928d2779" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43343 } }
I20260620 21:45:46.510026 6944 raft_consensus.cc:388] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:46.510092 6944 raft_consensus.cc:482] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:46.510183 6944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:46.511304 6944 raft_consensus.cc:504] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a362ed19dc29430fa1932aea928d2779" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43343 } }
I20260620 21:45:46.511623 6944 leader_election.cc:302] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a362ed19dc29430fa1932aea928d2779; no voters:
I20260620 21:45:46.511866 6944 leader_election.cc:288] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:46.511982 6949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:46.512249 6949 raft_consensus.cc:686] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [term 1 LEADER]: Becoming Leader. State: Replica: a362ed19dc29430fa1932aea928d2779, State: Running, Role: LEADER
I20260620 21:45:46.512579 6949 consensus_queue.cc:237] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a362ed19dc29430fa1932aea928d2779" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43343 } }
I20260620 21:45:46.513057 6944 sys_catalog.cc:564] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:46.514168 6951 sys_catalog.cc:455] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a362ed19dc29430fa1932aea928d2779. Latest consensus state: current_term: 1 leader_uuid: "a362ed19dc29430fa1932aea928d2779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a362ed19dc29430fa1932aea928d2779" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43343 } } }
I20260620 21:45:46.514294 6951 sys_catalog.cc:458] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:46.514668 6958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:46.515761 6950 sys_catalog.cc:455] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a362ed19dc29430fa1932aea928d2779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a362ed19dc29430fa1932aea928d2779" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43343 } } }
I20260620 21:45:46.515870 6950 sys_catalog.cc:458] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779 [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:46.517802 6958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:46.523523 6958 catalog_manager.cc:1347] Generated new cluster ID: a92ee0f358fc4b9792397aa9d93e9398
I20260620 21:45:46.523591 6958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:46.529915 6958 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:46.531080 6958 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:46.535064 6958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a362ed19dc29430fa1932aea928d2779: Generated new TSK 0
I20260620 21:45:46.535801 6958 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 21:45:46.562304 6974 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 21:45:46.562402 6974 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 21:45:46.563160 6947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:46.547390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36230 (local address 127.1.155.190:43343)
0620 21:45:46.547514 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:46.547518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:46.547699 (+ 181us) server_negotiation.cc:407] Connection header received
0620 21:45:46.547825 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:46.547828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:46.547882 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:46.547994 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:45:46.548220 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.549013 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.549883 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.550239 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.550341 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:46.550594 (+ 253us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:45:46.562716 (+ 12122us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45729/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 21:45:46.563013 (+ 297us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45729/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 21:45:46.563226 6947 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45729/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 21:45:46.563230 6973 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:45:46.547315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:43343 (local address 127.0.0.1:36230)
0620 21:45:46.547619 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:46.547632 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:46.547724 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:46.548015 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:46.548018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:46.548025 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:45:46.548116 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:46.548120 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.549145 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.549148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:46.549757 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:46.549763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.550345 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.550349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:46.550411 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:46.550436 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:45:46.562924 (+ 12488us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45729/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 21:45:46.563077 (+ 153us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.155.190:43343: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45729/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":148,"threads_started":1}
I20260620 21:45:46.563952 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 6877
2026-06-20T21:45:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (297 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 21:45:46.603410 1646 mini_oidc.cc:152] Starting JWKS server
I20260620 21:45:46.604282 1646 webserver.cc:460] Webserver started at https://127.0.0.1:41353/ using document root <none> and password file <none>
I20260620 21:45:46.604417 1646 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 21:45:46.604688 1646 webserver.cc:460] Webserver started at http://127.0.0.1:37245/ using document root <none> and password file <none>
2026-06-20T21:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:46Z Disabled control of system clock
I20260620 21:45:46.613154 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:35509
--webserver_interface=127.1.155.190
--webserver_port=0
--builtin_ntp_servers=127.1.155.148:39773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:35509
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41353/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 21:45:46.722126 6980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:46.722404 6980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:46.722482 6980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:46.726531 6980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:46.726585 6980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:46.726601 6980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:46.726619 6980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:46.726670 6980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:45:46.726704 6980 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:45:46.726733 6980 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41353/jwks.json
I20260620 21:45:46.731504 6980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:35509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:35509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41353/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:46.732575 6980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:46.734082 6980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:45:46.742419 6980 server_base.cc:1034] running on GCE node
W20260620 21:45:46.742435 6986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:46.742410 6988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:46.742415 6985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:46.743139 6980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:46.744210 6980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:46.745401 6980 hybrid_clock.cc:648] HybridClock initialized: now 1781991946745372 us; error 45 us; skew 500 ppm
I20260620 21:45:46.747251 6980 webserver.cc:460] Webserver started at http://127.1.155.190:43487/ using document root <none> and password file <none>
I20260620 21:45:46.747855 6980 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:46.747938 6980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:46.748150 6980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:46.750197 6980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "5a3a4d9f05df423cacb64c8c0a6656e6"
format_stamp: "Formatted at 2026-06-20 21:45:46 on dist-test-slave-5bvn"
server_key: "9c4100e749cc59a9bcca301472f80aeb"
server_key_iv: "d8316f921728e2fe04d7a817aa3b6ef7"
server_key_version: "encryptionkey@0"
I20260620 21:45:46.750736 6980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "5a3a4d9f05df423cacb64c8c0a6656e6"
format_stamp: "Formatted at 2026-06-20 21:45:46 on dist-test-slave-5bvn"
server_key: "9c4100e749cc59a9bcca301472f80aeb"
server_key_iv: "d8316f921728e2fe04d7a817aa3b6ef7"
server_key_version: "encryptionkey@0"
I20260620 21:45:46.754202 6980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:45:46.756508 6994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:46.757668 6980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:45:46.757797 6980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "5a3a4d9f05df423cacb64c8c0a6656e6"
format_stamp: "Formatted at 2026-06-20 21:45:46 on dist-test-slave-5bvn"
server_key: "9c4100e749cc59a9bcca301472f80aeb"
server_key_iv: "d8316f921728e2fe04d7a817aa3b6ef7"
server_key_version: "encryptionkey@0"
I20260620 21:45:46.757966 6980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:46.800612 6980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:46.801329 6980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:46.801545 6980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:46.808451 6980 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:35509
I20260620 21:45:46.808467 7046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:35509 every 8 connection(s)
I20260620 21:45:46.809463 6980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:46.812530 7047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:46.818010 7047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6: Bootstrap starting.
I20260620 21:45:46.818956 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 6980
I20260620 21:45:46.819070 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:46.819387 1646 external_mini_cluster.cc:1428] Setting key b66b2acd63e6738396e01a3e58d220c1
I20260620 21:45:46.820886 7047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:46.821605 7047 log.cc:824] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:46.823415 7047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6: No bootstrap required, opened a new log
I20260620 21:45:46.826800 7047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3a4d9f05df423cacb64c8c0a6656e6" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35509 } }
I20260620 21:45:46.827039 7047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:46.827095 7047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3a4d9f05df423cacb64c8c0a6656e6, State: Initialized, Role: FOLLOWER
I20260620 21:45:46.827569 7047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3a4d9f05df423cacb64c8c0a6656e6" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35509 } }
I20260620 21:45:46.827735 7047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:46.827809 7047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:46.827924 7047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:46.828227 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:46.820454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.1.155.190:35509)
0620 21:45:46.821122 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:46.821138 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:46.821173 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:45:46.821891 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:46.821909 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:46.822241 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:46.822571 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:46.822969 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.823949 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.824811 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.825298 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.825459 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:46.826331 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:46.826338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:46.826356 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:46.826748 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:46.826803 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:46.826823 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:46.826930 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:46.827045 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 21:45:46.827310 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":131,"threads_started":1}
I20260620 21:45:46.829149 7047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3a4d9f05df423cacb64c8c0a6656e6" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35509 } }
I20260620 21:45:46.829550 7047 leader_election.cc:302] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a3a4d9f05df423cacb64c8c0a6656e6; no voters:
I20260620 21:45:46.829936 7047 leader_election.cc:288] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:46.830013 7052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:46.830199 7052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [term 1 LEADER]: Becoming Leader. State: Replica: 5a3a4d9f05df423cacb64c8c0a6656e6, State: Running, Role: LEADER
I20260620 21:45:46.830492 7052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3a4d9f05df423cacb64c8c0a6656e6" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35509 } }
I20260620 21:45:46.831185 7047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:46.831426 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:45:46.831645 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a3a4d9f05df423cacb64c8c0a6656e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3a4d9f05df423cacb64c8c0a6656e6" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35509 } } }
I20260620 21:45:46.831741 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:46.831919 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a3a4d9f05df423cacb64c8c0a6656e6. Latest consensus state: current_term: 1 leader_uuid: "5a3a4d9f05df423cacb64c8c0a6656e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3a4d9f05df423cacb64c8c0a6656e6" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35509 } } }
I20260620 21:45:46.832016 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6 [sys.catalog]: This master's current role is: LEADER
W20260620 21:45:46.836424 7067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:45:46.836519 7067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:45:46.836575 7056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:46.837420 7056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:46.842746 7056 catalog_manager.cc:1347] Generated new cluster ID: bc2ef0b8d58e46979f79c8bf01bd5151
I20260620 21:45:46.842821 7056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:46.858686 7056 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:46.859705 7056 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:46.868671 7056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a3a4d9f05df423cacb64c8c0a6656e6: Generated new TSK 0
I20260620 21:45:46.869442 7056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:45:46.883952 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:46.878536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.1.155.190:35509)
0620 21:45:46.878677 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:46.878682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:46.878873 (+ 191us) server_negotiation.cc:407] Connection header received
0620 21:45:46.879133 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:46.879136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:46.879187 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:46.879284 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:45:46.879560 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.880124 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.881392 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.881641 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.881726 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:46.882086 (+ 360us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:45:46.883573 (+ 1487us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41353/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 21:45:46.883809 (+ 236us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41353/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 21:45:46.884033 7050 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41353/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 21:45:46.885766 7076 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:45:46.878455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:35509 (local address 127.0.0.1:45906)
0620 21:45:46.878779 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:46.878794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:46.878912 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:46.879305 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:46.879308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:46.879317 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:45:46.879450 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:46.879457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.880233 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.880237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:46.881264 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:46.881274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:46.881763 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:46.881767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:46.881887 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:46.881918 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:45:46.883750 (+ 1832us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41353/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 21:45:46.885583 (+ 1833us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.155.190:35509: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41353/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":1620,"thread_start_us":124,"threads_started":1}
I20260620 21:45:46.886392 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 6980
E20260620 21:45:46.894520 7077 webserver.cc:533] Webserver: sslize failed:
E20260620 21:45:46.894604 7077 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T21:45:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (321 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T21:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:46Z Disabled control of system clock
I20260620 21:45:46.934046 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:35317
--webserver_interface=127.1.155.190
--webserver_port=0
--builtin_ntp_servers=127.1.155.148:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:35317
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/testchainca.cert with env {}
W20260620 21:45:47.047384 7081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:47.047688 7081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:47.047757 7081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:47.051812 7081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:47.051872 7081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:47.051889 7081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:47.051905 7081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:47.051923 7081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:45:47.051939 7081 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:45:47.052003 7081 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 21:45:47.057324 7081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:35317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:35317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:47.058342 7081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:47.059322 7081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:45:47.067593 7081 server_base.cc:1034] running on GCE node
W20260620 21:45:47.067621 7087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:47.067765 7086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:47.067860 7089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:47.068388 7081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:47.069428 7081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:47.070585 7081 hybrid_clock.cc:648] HybridClock initialized: now 1781991947070556 us; error 42 us; skew 500 ppm
I20260620 21:45:47.072372 7081 webserver.cc:460] Webserver started at http://127.1.155.190:43557/ using document root <none> and password file <none>
I20260620 21:45:47.072912 7081 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:47.072980 7081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:47.073206 7081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:47.075184 7081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "d460ce16d8904b489770c0ecf2a9b173"
format_stamp: "Formatted at 2026-06-20 21:45:47 on dist-test-slave-5bvn"
server_key: "a34911e55b292c6f5e963338fcaf3b2d"
server_key_iv: "dcb8f3f6d7588cf50d965709d5271e79"
server_key_version: "encryptionkey@0"
I20260620 21:45:47.075723 7081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "d460ce16d8904b489770c0ecf2a9b173"
format_stamp: "Formatted at 2026-06-20 21:45:47 on dist-test-slave-5bvn"
server_key: "a34911e55b292c6f5e963338fcaf3b2d"
server_key_iv: "dcb8f3f6d7588cf50d965709d5271e79"
server_key_version: "encryptionkey@0"
I20260620 21:45:47.079329 7081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:45:47.081941 7095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:47.083315 7081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:45:47.083459 7081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "d460ce16d8904b489770c0ecf2a9b173"
format_stamp: "Formatted at 2026-06-20 21:45:47 on dist-test-slave-5bvn"
server_key: "a34911e55b292c6f5e963338fcaf3b2d"
server_key_iv: "dcb8f3f6d7588cf50d965709d5271e79"
server_key_version: "encryptionkey@0"
I20260620 21:45:47.083590 7081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:47.104005 7081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:47.105029 7081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:47.105266 7081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:47.113087 7081 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:35317
I20260620 21:45:47.113147 7147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:35317 every 8 connection(s)
I20260620 21:45:47.114126 7081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:47.117729 7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:47.122030 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 7081
I20260620 21:45:47.122268 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:47.122611 1646 external_mini_cluster.cc:1428] Setting key 89633bcf7103064574bc1912d6851107
I20260620 21:45:47.125176 7148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173: Bootstrap starting.
I20260620 21:45:47.127990 7148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:47.128880 7148 log.cc:824] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:47.131184 7148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173: No bootstrap required, opened a new log
I20260620 21:45:47.134361 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:47.124486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.1.155.190:35317)
0620 21:45:47.125196 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:47.125219 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:47.125264 (+ 45us) server_negotiation.cc:407] Connection header received
0620 21:45:47.126152 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:47.126174 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:47.126704 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:47.127063 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:47.127512 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:47.128988 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:47.130265 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:47.130949 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:47.131118 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:47.131947 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:47.131954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:47.131969 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:45:47.132461 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:47.132529 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:45:47.132555 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:47.132652 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:47.132779 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 21:45:47.133042 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":141,"threads_started":1}
I20260620 21:45:47.135141 7148 raft_consensus.cc:348] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d460ce16d8904b489770c0ecf2a9b173" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35317 } }
I20260620 21:45:47.135370 7148 raft_consensus.cc:374] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:47.135437 7148 raft_consensus.cc:729] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d460ce16d8904b489770c0ecf2a9b173, State: Initialized, Role: FOLLOWER
I20260620 21:45:47.135840 7148 consensus_queue.cc:260] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d460ce16d8904b489770c0ecf2a9b173" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35317 } }
I20260620 21:45:47.135989 7148 raft_consensus.cc:388] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:47.136070 7148 raft_consensus.cc:482] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:47.136188 7148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:47.137148 7148 raft_consensus.cc:504] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d460ce16d8904b489770c0ecf2a9b173" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35317 } }
I20260620 21:45:47.137444 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:45:47.137504 7148 leader_election.cc:302] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d460ce16d8904b489770c0ecf2a9b173; no voters:
I20260620 21:45:47.137807 7148 leader_election.cc:288] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:47.137974 7153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:47.138226 7153 raft_consensus.cc:686] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [term 1 LEADER]: Becoming Leader. State: Replica: d460ce16d8904b489770c0ecf2a9b173, State: Running, Role: LEADER
I20260620 21:45:47.138602 7153 consensus_queue.cc:237] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d460ce16d8904b489770c0ecf2a9b173" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35317 } }
I20260620 21:45:47.139089 7148 sys_catalog.cc:564] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:47.140389 7155 sys_catalog.cc:455] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d460ce16d8904b489770c0ecf2a9b173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d460ce16d8904b489770c0ecf2a9b173" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35317 } } }
I20260620 21:45:47.140412 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d460ce16d8904b489770c0ecf2a9b173. Latest consensus state: current_term: 1 leader_uuid: "d460ce16d8904b489770c0ecf2a9b173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d460ce16d8904b489770c0ecf2a9b173" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35317 } } }
I20260620 21:45:47.140522 7155 sys_catalog.cc:458] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:47.140522 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173 [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:47.140913 7164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:47.143595 7164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:47.148722 7164 catalog_manager.cc:1347] Generated new cluster ID: 6e024b4befad4912889ecaba5b874bf9
I20260620 21:45:47.148787 7164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:47.173229 7164 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:47.174247 7164 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:47.182807 7164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d460ce16d8904b489770c0ecf2a9b173: Generated new TSK 0
I20260620 21:45:47.183463 7164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:45:47.669679 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:47.199716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52404 (local address 127.1.155.190:35317)
0620 21:45:47.199861 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:47.199867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:47.200120 (+ 253us) server_negotiation.cc:407] Connection header received
0620 21:45:47.200199 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:47.200202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:47.200248 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:47.200344 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:45:47.200588 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:47.201121 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:47.202350 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:47.202574 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:47.202654 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:47.202999 (+ 345us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:45:47.669307 (+466308us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 21:45:47.669558 (+ 251us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52404: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 21:45:47.669798 7151 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52404: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 21:45:47.669958 7177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:45:47.199642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:35317 (local address 127.0.0.1:52404)
0620 21:45:47.199923 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:47.199940 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:47.200065 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:47.200362 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:47.200366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:47.200373 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:45:47.200484 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:47.200490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:47.201220 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:47.201224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:47.202237 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:47.202248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:47.202685 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:47.202689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:47.202784 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:47.202814 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:45:47.669545 (+466731us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 21:45:47.669743 (+ 198us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.155.190:35317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":100,"threads_started":1}
I20260620 21:45:47.670512 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 7081
2026-06-20T21:45:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (764 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (120 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:45:47 dist-test-slave-5bvn krb5kdc[7189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:45:47 dist-test-slave-5bvn krb5kdc[7189](info): set up 2 sockets
Jun 20 21:45:47 dist-test-slave-5bvn krb5kdc[7189](info): commencing operation
krb5kdc: starting...
W20260620 21:45:48.950205 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:45:49 dist-test-slave-5bvn krb5kdc[7189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:45:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:45:49.073529 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:38635
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:44029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:38635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:49.186846 7202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:49.187188 7202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:49.187268 7202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:49.191509 7202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:49.191603 7202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:49.191643 7202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:49.191696 7202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:49.191730 7202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:45:49.198225 7202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:44029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:38635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:38635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:49.199388 7202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:49.200428 7202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:49.209123 7210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:49.209275 7202 server_base.cc:1034] running on GCE node
W20260620 21:45:49.209154 7208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:49.209204 7207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:49.210019 7202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:49.211122 7202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:49.212320 7202 hybrid_clock.cc:648] HybridClock initialized: now 1781991949212293 us; error 42 us; skew 500 ppm
Jun 20 21:45:49 dist-test-slave-5bvn krb5kdc[7189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991949, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:49.218721 7202 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:45:49.219692 7202 webserver.cc:460] Webserver started at http://127.1.155.190:42687/ using document root <none> and password file <none>
I20260620 21:45:49.220254 7202 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:49.220343 7202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:49.220564 7202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:49.222690 7202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "66b119f5b36d430987f951d3640dd3ef"
format_stamp: "Formatted at 2026-06-20 21:45:49 on dist-test-slave-5bvn"
server_key: "cd8b677d0bbe8e5541704439299fb316"
server_key_iv: "b4067a50c0f4ea4d3dc37ed58001c63d"
server_key_version: "encryptionkey@0"
I20260620 21:45:49.223255 7202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "66b119f5b36d430987f951d3640dd3ef"
format_stamp: "Formatted at 2026-06-20 21:45:49 on dist-test-slave-5bvn"
server_key: "cd8b677d0bbe8e5541704439299fb316"
server_key_iv: "b4067a50c0f4ea4d3dc37ed58001c63d"
server_key_version: "encryptionkey@0"
I20260620 21:45:49.227048 7202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 21:45:49.230201 7217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:49.231666 7202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:45:49.231851 7202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "66b119f5b36d430987f951d3640dd3ef"
format_stamp: "Formatted at 2026-06-20 21:45:49 on dist-test-slave-5bvn"
server_key: "cd8b677d0bbe8e5541704439299fb316"
server_key_iv: "b4067a50c0f4ea4d3dc37ed58001c63d"
server_key_version: "encryptionkey@0"
I20260620 21:45:49.231979 7202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:49.248773 7202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:49.252691 7202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:49.252903 7202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:49.260044 7202 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:38635
I20260620 21:45:49.260071 7269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:38635 every 8 connection(s)
I20260620 21:45:49.260982 7202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:49.263664 7270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:49.268949 7270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef: Bootstrap starting.
I20260620 21:45:49.270349 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 7202
I20260620 21:45:49.270447 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:49.270869 1646 external_mini_cluster.cc:1428] Setting key e7a14d572194a47f6b5a6e1303b5993c
I20260620 21:45:49.271822 7270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:49.272760 7270 log.cc:824] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:49.274931 7270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef: No bootstrap required, opened a new log
I20260620 21:45:49.278959 7270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b119f5b36d430987f951d3640dd3ef" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38635 } }
I20260620 21:45:49.279165 7270 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:49.279223 7270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b119f5b36d430987f951d3640dd3ef, State: Initialized, Role: FOLLOWER
I20260620 21:45:49.279665 7270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b119f5b36d430987f951d3640dd3ef" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38635 } }
I20260620 21:45:49.279827 7270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:49.279911 7270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Jun 20 21:45:49 dist-test-slave-5bvn krb5kdc[7189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:49.280026 7270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:49.281060 7270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b119f5b36d430987f951d3640dd3ef" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38635 } }
I20260620 21:45:49.281414 7270 leader_election.cc:302] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66b119f5b36d430987f951d3640dd3ef; no voters:
I20260620 21:45:49.281741 7270 leader_election.cc:288] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:49.281858 7275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:49.282083 7275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [term 1 LEADER]: Becoming Leader. State: Replica: 66b119f5b36d430987f951d3640dd3ef, State: Running, Role: LEADER
I20260620 21:45:49.282433 7275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b119f5b36d430987f951d3640dd3ef" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38635 } }
I20260620 21:45:49.283140 7270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:49.284032 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b119f5b36d430987f951d3640dd3ef. Latest consensus state: current_term: 1 leader_uuid: "66b119f5b36d430987f951d3640dd3ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b119f5b36d430987f951d3640dd3ef" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38635 } } }
I20260620 21:45:49.284142 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:49.284507 7284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:49.284507 7276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66b119f5b36d430987f951d3640dd3ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b119f5b36d430987f951d3640dd3ef" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38635 } } }
I20260620 21:45:49.284667 7276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:49.286513 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:49.271969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.1.155.190:38635)
0620 21:45:49.272416 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:49.272434 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:49.272475 (+ 41us) server_negotiation.cc:407] Connection header received
0620 21:45:49.273302 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:49.273322 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:49.273709 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:49.274077 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:49.274752 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:49.276170 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:49.277070 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:49.277856 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:49.278050 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:49.280675 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:49.280680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:49.280692 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:49.280718 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:49.282720 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:49.283236 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:49.283242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:49.283250 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:49.283381 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:49.283683 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:49.283686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:49.283688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:49.284284 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:49.285059 (+ 775us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:49.285223 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 21:45:49.285509 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":122,"threads_started":1}
I20260620 21:45:49.288138 7284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:49.293661 7284 catalog_manager.cc:1347] Generated new cluster ID: cba49176a66c4861894363be2d8ff915
I20260620 21:45:49.293748 7284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:49.321166 7284 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:49.322203 7284 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:49.330585 7284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66b119f5b36d430987f951d3640dd3ef: Generated new TSK 0
I20260620 21:45:49.331261 7284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:45:49.365229 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:38635
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:44029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:49.473181 7296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:49.473455 7296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:49.473523 7296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:49.477517 7296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:49.477602 7296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:49.477690 7296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:45:49.482914 7296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:44029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:38635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:49.483994 7296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:49.485036 7296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:49.494321 7302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:49.494383 7301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:49.494374 7304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:49.494880 7296 server_base.cc:1034] running on GCE node
I20260620 21:45:49.495285 7296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:49.495832 7296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:49.497008 7296 hybrid_clock.cc:648] HybridClock initialized: now 1781991949496974 us; error 48 us; skew 500 ppm
Jun 20 21:45:49 dist-test-slave-5bvn krb5kdc[7189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991949, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:49.499675 7296 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:45:49.500483 7296 webserver.cc:460] Webserver started at http://127.1.155.129:41865/ using document root <none> and password file <none>
I20260620 21:45:49.501012 7296 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:49.501102 7296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:49.501322 7296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:49.503297 7296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "5e0d915024564fa6ac4069d4b998ccb2"
format_stamp: "Formatted at 2026-06-20 21:45:49 on dist-test-slave-5bvn"
server_key: "5602c1babaec9469c6ac5acc06f26ad9"
server_key_iv: "208e0f9fe726c626519b65d762193685"
server_key_version: "encryptionkey@0"
I20260620 21:45:49.503834 7296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "5e0d915024564fa6ac4069d4b998ccb2"
format_stamp: "Formatted at 2026-06-20 21:45:49 on dist-test-slave-5bvn"
server_key: "5602c1babaec9469c6ac5acc06f26ad9"
server_key_iv: "208e0f9fe726c626519b65d762193685"
server_key_version: "encryptionkey@0"
I20260620 21:45:49.507344 7296 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:45:49.509769 7311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:49.511277 7296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:45:49.511416 7296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "5e0d915024564fa6ac4069d4b998ccb2"
format_stamp: "Formatted at 2026-06-20 21:45:49 on dist-test-slave-5bvn"
server_key: "5602c1babaec9469c6ac5acc06f26ad9"
server_key_iv: "208e0f9fe726c626519b65d762193685"
server_key_version: "encryptionkey@0"
I20260620 21:45:49.511528 7296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:49.522223 7296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:49.525305 7296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:49.525501 7296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:49.526119 7296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:49.526909 7296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:49.526983 7296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:49.527065 7296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:49.527107 7296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:49.535965 7296 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:34049
I20260620 21:45:49.535986 7424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:34049 every 8 connection(s)
I20260620 21:45:49.536906 7296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:45:49.541540 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 7296
I20260620 21:45:49.541680 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:45:49.542001 1646 external_mini_cluster.cc:1428] Setting key 7c28eb9090c6be43ec8670e62cd840f3
Jun 20 21:45:49 dist-test-slave-5bvn krb5kdc[7189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991949, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
I20260620 21:45:49.553977 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:49.539535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:47503 (local address 127.1.155.190:38635)
0620 21:45:49.539757 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:49.539761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:49.540584 (+ 823us) server_negotiation.cc:407] Connection header received
0620 21:45:49.542479 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:49.542483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:49.542590 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:49.542691 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:49.544271 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:49.544840 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:49.546198 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:49.546482 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:49.546574 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:49.550097 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:49.550101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:49.550103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:49.550131 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:49.551693 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:49.552281 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:49.552286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:49.552287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:49.552371 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:49.552783 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:49.552787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:49.552788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:49.552979 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:49.553063 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:49.553681 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 21:45:49.553791 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
I20260620 21:45:49.555126 7427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:49.539958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:38635 (local address 127.1.155.129:47503)
0620 21:45:49.540426 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:49.540470 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:49.542197 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:49.542915 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:49.542923 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:49.543381 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:49.544083 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:49.544109 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:49.544979 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:49.544985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:49.546069 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:49.546079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:49.546744 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:49.546756 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:49.547023 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:49.547789 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:49.547814 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:49.549893 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:49.551839 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:49.551846 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:49.551864 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:49.552158 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:49.552473 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:49.552477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:49.552479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:49.552662 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:49.553082 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:49.553089 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:49.553364 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 21:45:49.553607 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 21:45:49.553973 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":115,"threads_started":1}
I20260620 21:45:49.556649 7425 heartbeater.cc:347] Connected to a master server at 127.1.155.190:38635
I20260620 21:45:49.556928 7425 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:49.557440 7425 heartbeater.cc:511] Master 127.1.155.190:38635 requested a full tablet report, sending...
I20260620 21:45:49.559327 7234 ts_manager.cc:194] Registered new tserver with Master: 5e0d915024564fa6ac4069d4b998ccb2 (127.1.155.129:34049)
I20260620 21:45:49.561028 7234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:47503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:45:49.573778 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:38635
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:44029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:49.687232 7430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:49.687564 7430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:49.687641 7430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:49.691913 7430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:49.692018 7430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:49.692144 7430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:45:49.698099 7430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:44029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:38635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:49.699158 7430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:49.700139 7430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:49.709712 7436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:49.709846 7430 server_base.cc:1034] running on GCE node
W20260620 21:45:49.709764 7435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:49.709707 7438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:49.710431 7430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:49.711117 7430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:49.712327 7430 hybrid_clock.cc:648] HybridClock initialized: now 1781991949712290 us; error 49 us; skew 500 ppm
Jun 20 21:45:49 dist-test-slave-5bvn krb5kdc[7189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991949, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:49.715360 7430 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:45:49.716321 7430 webserver.cc:460] Webserver started at http://127.1.155.130:38195/ using document root <none> and password file <none>
I20260620 21:45:49.716877 7430 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:49.716960 7430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:49.717186 7430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:49.719326 7430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "0b01d6ca99b74002b37bb1898340e396"
format_stamp: "Formatted at 2026-06-20 21:45:49 on dist-test-slave-5bvn"
server_key: "92afec3b153462537358fb09f20577dc"
server_key_iv: "3fee85043edd88c69bf2afc29cfb944d"
server_key_version: "encryptionkey@0"
I20260620 21:45:49.719906 7430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "0b01d6ca99b74002b37bb1898340e396"
format_stamp: "Formatted at 2026-06-20 21:45:49 on dist-test-slave-5bvn"
server_key: "92afec3b153462537358fb09f20577dc"
server_key_iv: "3fee85043edd88c69bf2afc29cfb944d"
server_key_version: "encryptionkey@0"
I20260620 21:45:49.723636 7430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 21:45:49.726024 7445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:49.727139 7430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:45:49.727264 7430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "0b01d6ca99b74002b37bb1898340e396"
format_stamp: "Formatted at 2026-06-20 21:45:49 on dist-test-slave-5bvn"
server_key: "92afec3b153462537358fb09f20577dc"
server_key_iv: "3fee85043edd88c69bf2afc29cfb944d"
server_key_version: "encryptionkey@0"
I20260620 21:45:49.727372 7430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:49.739017 7430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:49.742352 7430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:49.742550 7430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:49.743124 7430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:49.743902 7430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:49.743971 7430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:49.744038 7430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:49.744083 7430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:49.753044 7430 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:36537
I20260620 21:45:49.753073 7558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:36537 every 8 connection(s)
I20260620 21:45:49.754006 7430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:45:49.760237 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 7430
I20260620 21:45:49.760385 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:45:49.760679 1646 external_mini_cluster.cc:1428] Setting key b885c6113f1e48795972d123d82f5df6
Jun 20 21:45:49 dist-test-slave-5bvn krb5kdc[7189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991949, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:49.769241 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:49.755886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:54507 (local address 127.1.155.190:38635)
0620 21:45:49.756017 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:49.756020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:49.756840 (+ 820us) server_negotiation.cc:407] Connection header received
0620 21:45:49.758268 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:49.758272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:49.758329 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:49.758410 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:49.759541 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:49.760091 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:49.761505 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:49.761733 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:49.761866 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:49.765115 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:49.765122 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:49.765128 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:49.765164 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:49.766968 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:49.767499 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:49.767505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:49.767508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:49.767628 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:49.767975 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:49.767981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:49.767985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:49.768261 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:49.768382 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:49.768925 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 21:45:49.769056 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:45:49.770094 7561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:49.756247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:38635 (local address 127.1.155.130:54507)
0620 21:45:49.756691 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:49.756732 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:49.758056 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:49.758569 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:49.758576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:49.758927 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:49.759403 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:49.759417 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:49.760216 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:49.760225 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:49.761388 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:49.761398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:49.761905 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:49.761910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:49.762108 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:49.762734 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:49.762760 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:49.764836 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:49.767090 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:49.767095 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:49.767109 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:49.767385 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:49.767736 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:49.767738 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:49.767740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:49.767883 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:49.768363 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:49.768369 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:49.768619 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 21:45:49.768832 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 21:45:49.769098 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
I20260620 21:45:49.771301 7559 heartbeater.cc:347] Connected to a master server at 127.1.155.190:38635
I20260620 21:45:49.771539 7559 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:49.772058 7559 heartbeater.cc:511] Master 127.1.155.190:38635 requested a full tablet report, sending...
I20260620 21:45:49.773108 7234 ts_manager.cc:194] Registered new tserver with Master: 0b01d6ca99b74002b37bb1898340e396 (127.1.155.130:36537)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
I20260620 21:45:49.773950 7234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:54507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:45:49.789448 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:38635
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:44029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:49.903231 7564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:49.903527 7564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:49.903574 7564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:49.907998 7564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:49.908069 7564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:49.908188 7564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:45:49.913648 7564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:44029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:38635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:49.915032 7564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:49.916344 7564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:49.927168 7570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:49.927196 7569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:49.927582 7564 server_base.cc:1034] running on GCE node
W20260620 21:45:49.927194 7572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:49.928103 7564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:49.928809 7564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:49.929934 7564 hybrid_clock.cc:648] HybridClock initialized: now 1781991949929910 us; error 31 us; skew 500 ppm
Jun 20 21:45:49 dist-test-slave-5bvn krb5kdc[7189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991949, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:49.933408 7564 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:45:49.934392 7564 webserver.cc:460] Webserver started at http://127.1.155.131:36751/ using document root <none> and password file <none>
I20260620 21:45:49.934989 7564 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:49.935052 7564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:49.935323 7564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:49.937492 7564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "fa4d8c181d91441fb68c20135bbbcd9e"
format_stamp: "Formatted at 2026-06-20 21:45:49 on dist-test-slave-5bvn"
server_key: "ef553b116f8c8fd1587ed00f7feaf287"
server_key_iv: "8182cf83c3fdb7c742b3f494cd989c08"
server_key_version: "encryptionkey@0"
I20260620 21:45:49.938089 7564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "fa4d8c181d91441fb68c20135bbbcd9e"
format_stamp: "Formatted at 2026-06-20 21:45:49 on dist-test-slave-5bvn"
server_key: "ef553b116f8c8fd1587ed00f7feaf287"
server_key_iv: "8182cf83c3fdb7c742b3f494cd989c08"
server_key_version: "encryptionkey@0"
I20260620 21:45:49.942030 7564 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:45:49.944558 7579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:49.946094 7564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:45:49.946237 7564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "fa4d8c181d91441fb68c20135bbbcd9e"
format_stamp: "Formatted at 2026-06-20 21:45:49 on dist-test-slave-5bvn"
server_key: "ef553b116f8c8fd1587ed00f7feaf287"
server_key_iv: "8182cf83c3fdb7c742b3f494cd989c08"
server_key_version: "encryptionkey@0"
I20260620 21:45:49.946344 7564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:49.963884 7564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:49.967355 7564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:49.967557 7564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:49.968149 7564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:49.968955 7564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:49.969000 7564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:49.969070 7564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:49.969108 7564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:49.980386 7564 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:45351
I20260620 21:45:49.980499 7692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:45351 every 8 connection(s)
I20260620 21:45:49.981460 7564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:45:49.986583 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 7564
I20260620 21:45:49.986691 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:45:49.986989 1646 external_mini_cluster.cc:1428] Setting key c57f113b45a6a5fb7254fa2555c0d8ad
Jun 20 21:45:49 dist-test-slave-5bvn krb5kdc[7189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991949, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:45:49.996389 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:49.983511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:56901 (local address 127.1.155.190:38635)
0620 21:45:49.983701 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:49.983706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:49.984574 (+ 868us) server_negotiation.cc:407] Connection header received
0620 21:45:49.986029 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:49.986033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:49.986084 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:49.986169 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:49.987718 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:49.988476 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:49.990230 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:49.990444 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:49.990538 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:49.993066 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:49.993069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:49.993071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:49.993095 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:49.994537 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:49.995028 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:49.995031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:49.995032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:49.995108 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:49.995403 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:49.995405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:49.995406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:49.995571 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:49.995646 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:49.996159 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 21:45:49.996267 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 21:45:49.997227 7695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:49.983874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:38635 (local address 127.1.155.131:56901)
0620 21:45:49.984380 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:49.984424 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:49.985784 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:49.986325 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:49.986331 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:49.986920 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:49.987566 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:49.987585 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:49.988648 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:49.988652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:49.990097 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:49.990108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:49.990626 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:49.990630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:49.990786 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:49.991428 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:49.991445 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:49.992899 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:49.994662 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:49.994668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:49.994682 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:49.994938 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:49.995189 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:49.995191 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:49.995193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:49.995323 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:49.995683 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:49.995689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:49.995921 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 21:45:49.996097 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 21:45:49.996357 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":112,"threads_started":1}
I20260620 21:45:49.998796 7693 heartbeater.cc:347] Connected to a master server at 127.1.155.190:38635
I20260620 21:45:49.999063 7693 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:49.999577 7693 heartbeater.cc:511] Master 127.1.155.190:38635 requested a full tablet report, sending...
I20260620 21:45:50.000784 7234 ts_manager.cc:194] Registered new tserver with Master: fa4d8c181d91441fb68c20135bbbcd9e (127.1.155.131:45351)
I20260620 21:45:50.001497 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:45:50.001480 7234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:56901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/test-admin.keytab.
Jun 20 21:45:50 dist-test-slave-5bvn krb5kdc[7189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:50.019228 1646 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 21:45:50.022625 7700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:45:50.021123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.1.155.129:34049)
0620 21:45:50.021486 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:50.021501 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:50.021525 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:45:50.021574 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:50.021579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:50.021770 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:50.021970 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:50.022448 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42592: BlockingRecv error: recv got EOF from 127.0.0.1:42592 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":145,"threads_started":1}
W20260620 21:45:50.022678 1646 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/krb5cc' of type FILE
Jun 20 21:45:50 dist-test-slave-5bvn krb5kdc[7189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:50.024006 1646 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 21:45:50 dist-test-slave-5bvn krb5kdc[7189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:50.024887 1646 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 21:45:50 dist-test-slave-5bvn krb5kdc[7189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.129@KRBTEST.COM
I20260620 21:45:50.036155 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:50.025762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42608 (local address 127.1.155.129:34049)
0620 21:45:50.025912 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:50.025917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:50.026010 (+ 93us) server_negotiation.cc:407] Connection header received
0620 21:45:50.026159 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:50.026162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:50.026210 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:50.026295 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:50.026855 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:50.027621 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:50.028448 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:50.028836 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:50.028959 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:50.031347 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:50.031356 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:50.031369 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:50.031392 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:50.033516 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:50.034156 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:50.034167 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:50.034169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:50.034252 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:50.034584 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:50.034587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:50.034590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:50.035078 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:50.035371 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:50.035665 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 21:45:50.035889 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 21:45:50.040290 7700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:45:50.039097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.1.155.129:34049)
0620 21:45:50.039285 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:50.039290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:50.039420 (+ 130us) server_negotiation.cc:407] Connection header received
0620 21:45:50.039580 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:50.039583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:50.039629 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:50.039730 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:50.040173 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42610: BlockingRecv error: recv got EOF from 127.0.0.1:42610 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260620 21:45:50.040504 1646 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/krb5cc' of type FILE
Jun 20 21:45:50 dist-test-slave-5bvn krb5kdc[7189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:50.041802 1646 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 21:45:50 dist-test-slave-5bvn krb5kdc[7189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:50.042819 1646 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 21:45:50 dist-test-slave-5bvn krb5kdc[7189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.129@KRBTEST.COM
I20260620 21:45:50.052101 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:50.043700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42614 (local address 127.1.155.129:34049)
0620 21:45:50.043851 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:50.043855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:50.043968 (+ 113us) server_negotiation.cc:407] Connection header received
0620 21:45:50.044115 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:50.044118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:50.044170 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:50.044259 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:50.044784 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:50.045323 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:50.046195 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:50.046444 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:50.046558 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:50.048938 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:50.048942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:50.048944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:50.048968 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:50.050458 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:50.050883 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:50.050887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:50.050889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:50.050989 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:50.051270 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:50.051273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:50.051275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:50.051446 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:50.051580 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:50.051825 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 21:45:50.051937 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 21:45:50.054909 7700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:45:50.053949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42624 (local address 127.1.155.129:34049)
0620 21:45:50.054125 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:50.054128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:50.054164 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:45:50.054273 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:50.054275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:50.054311 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:50.054370 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:50.054794 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42624: BlockingRecv error: recv got EOF from 127.0.0.1:42624 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 21:45:50.054942 1646 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781991920067886-1646-0/krb5kdc/krb5cc' of type FILE
Jun 20 21:45:50 dist-test-slave-5bvn krb5kdc[7189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:50.056001 1646 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 21:45:50 dist-test-slave-5bvn krb5kdc[7189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:50.056948 1646 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 21:45:50 dist-test-slave-5bvn krb5kdc[7189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.129@KRBTEST.COM
I20260620 21:45:50.066191 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:50.058082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42628 (local address 127.1.155.129:34049)
0620 21:45:50.058198 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:50.058202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:50.058364 (+ 162us) server_negotiation.cc:407] Connection header received
0620 21:45:50.058496 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:50.058499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:50.058537 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:50.058628 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:50.059189 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:50.059690 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:50.060564 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:50.060797 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:50.060907 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:50.063277 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:50.063281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:50.063284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:50.063311 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:50.064566 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:50.064978 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:50.064983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:50.064985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:50.065065 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:50.065338 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:50.065341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:50.065343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:50.065513 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:50.065644 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:50.065917 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 21:45:50.066021 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:45:50.067281 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 7296
I20260620 21:45:50.073055 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 7430
I20260620 21:45:50.078286 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 7564
I20260620 21:45:50.083698 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 7202
2026-06-20T21:45:50Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2173 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:45:50 dist-test-slave-5bvn krb5kdc[7713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:45:50 dist-test-slave-5bvn krb5kdc[7713](info): set up 2 sockets
Jun 20 21:45:50 dist-test-slave-5bvn krb5kdc[7713](info): commencing operation
krb5kdc: starting...
W20260620 21:45:51.134294 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:45:51 dist-test-slave-5bvn krb5kdc[7713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:45:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.190.keytab.
Entry for principal oryx/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.190.keytab.
I20260620 21:45:51.253633 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:36873
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.190.keytab
--principal=oryx/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:36873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:51.363363 7726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:51.363646 7726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:51.363687 7726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:51.367870 7726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:51.367928 7726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:51.367947 7726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:51.367964 7726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:51.367980 7726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:45:51.373225 7726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:36873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.190.keytab
--principal=oryx/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:36873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:51.374264 7726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:51.375249 7726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:51.383358 7732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:51.383386 7734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:51.383412 7731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:51.383688 7726 server_base.cc:1034] running on GCE node
I20260620 21:45:51.384160 7726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:51.385179 7726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:51.386359 7726 hybrid_clock.cc:648] HybridClock initialized: now 1781991951386347 us; error 26 us; skew 500 ppm
Jun 20 21:45:51 dist-test-slave-5bvn krb5kdc[7713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991951, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:51.388929 7726 init.cc:377] Logged in from keytab as oryx/127.1.155.190@KRBTEST.COM (short username oryx)
I20260620 21:45:51.389807 7726 webserver.cc:460] Webserver started at http://127.1.155.190:33649/ using document root <none> and password file <none>
I20260620 21:45:51.390398 7726 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:51.390488 7726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:51.390707 7726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:51.392635 7726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "32c4b8ae8381486abded8f30d55598ee"
format_stamp: "Formatted at 2026-06-20 21:45:51 on dist-test-slave-5bvn"
server_key: "09f8dcd9e1b00e1dcf50dbb5b9b9106d"
server_key_iv: "6b804b4fb7d302cff97aede114830d35"
server_key_version: "encryptionkey@0"
I20260620 21:45:51.393183 7726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "32c4b8ae8381486abded8f30d55598ee"
format_stamp: "Formatted at 2026-06-20 21:45:51 on dist-test-slave-5bvn"
server_key: "09f8dcd9e1b00e1dcf50dbb5b9b9106d"
server_key_iv: "6b804b4fb7d302cff97aede114830d35"
server_key_version: "encryptionkey@0"
I20260620 21:45:51.396837 7726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:45:51.399283 7741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:51.400496 7726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:45:51.400646 7726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "32c4b8ae8381486abded8f30d55598ee"
format_stamp: "Formatted at 2026-06-20 21:45:51 on dist-test-slave-5bvn"
server_key: "09f8dcd9e1b00e1dcf50dbb5b9b9106d"
server_key_iv: "6b804b4fb7d302cff97aede114830d35"
server_key_version: "encryptionkey@0"
I20260620 21:45:51.400758 7726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:51.429095 7726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:51.432194 7726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:51.432404 7726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:51.439679 7726 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:36873
I20260620 21:45:51.439980 7793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:36873 every 8 connection(s)
I20260620 21:45:51.440639 7726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:51.443400 7794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:51.448642 7794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee: Bootstrap starting.
I20260620 21:45:51.449867 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 7726
I20260620 21:45:51.450003 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:51.450290 1646 external_mini_cluster.cc:1428] Setting key 23d2f6f3cb9a2437e57af19f93933a47
I20260620 21:45:51.451538 7794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:51.452237 7794 log.cc:824] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:51.454373 7794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee: No bootstrap required, opened a new log
I20260620 21:45:51.458848 7794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } }
I20260620 21:45:51.459116 7794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Jun 20 21:45:51 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
I20260620 21:45:51.459211 7794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c4b8ae8381486abded8f30d55598ee, State: Initialized, Role: FOLLOWER
I20260620 21:45:51.459790 7794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } }
I20260620 21:45:51.460000 7794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:51.460075 7794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:51.460173 7794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:51.461094 7794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } }
I20260620 21:45:51.461436 7794 leader_election.cc:302] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32c4b8ae8381486abded8f30d55598ee; no voters:
I20260620 21:45:51.461792 7794 leader_election.cc:288] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:51.461891 7799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:51.462091 7799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [term 1 LEADER]: Becoming Leader. State: Replica: 32c4b8ae8381486abded8f30d55598ee, State: Running, Role: LEADER
I20260620 21:45:51.462450 7799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } }
I20260620 21:45:51.463128 7794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:51.464156 7800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32c4b8ae8381486abded8f30d55598ee. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } }
I20260620 21:45:51.464275 7800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:51.464646 7801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } }
I20260620 21:45:51.465265 7801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:51.466212 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:51.451348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46240 (local address 127.1.155.190:36873)
0620 21:45:51.451739 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:51.451754 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:51.451787 (+ 33us) server_negotiation.cc:407] Connection header received
0620 21:45:51.452410 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:51.452436 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:51.452826 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:51.453148 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:51.453980 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:51.455008 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:51.456146 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:51.456624 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:51.456742 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:51.459950 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:51.459955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:51.459968 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:51.460033 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:51.462127 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:51.462640 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:51.462646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:51.462652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:51.462770 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:51.463059 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:51.463064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:51.463065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:51.463575 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:51.464438 (+ 863us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:51.464584 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 21:45:51.464890 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":139,"threads_started":1}
W20260620 21:45:51.468945 7815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:45:51.469022 7815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:45:51.469154 7808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:51.469966 7808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:51.475666 7808 catalog_manager.cc:1347] Generated new cluster ID: 4a3ea318e1ba4c3598f9fdf369f79031
I20260620 21:45:51.475744 7808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:51.494448 7808 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:51.495362 7808 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:51.513370 7808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee: Generated new TSK 0
I20260620 21:45:51.514139 7808 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.129.keytab.
Entry for principal oryx/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.129.keytab.
I20260620 21:45:51.551242 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:36873
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.129.keytab
--principal=oryx/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:51.661099 7820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:51.661341 7820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:51.661383 7820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:51.666481 7820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:51.666566 7820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:51.666657 7820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:45:51.676764 7820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.129.keytab
--principal=oryx/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:36873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:51.678541 7820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:51.680078 7820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:51.691668 7828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:51.691669 7826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:51.691668 7825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:51.692232 7820 server_base.cc:1034] running on GCE node
I20260620 21:45:51.692641 7820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:51.693331 7820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:51.694547 7820 hybrid_clock.cc:648] HybridClock initialized: now 1781991951694529 us; error 32 us; skew 500 ppm
Jun 20 21:45:51 dist-test-slave-5bvn krb5kdc[7713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991951, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:51.698150 7820 init.cc:377] Logged in from keytab as oryx/127.1.155.129@KRBTEST.COM (short username oryx)
I20260620 21:45:51.699350 7820 webserver.cc:460] Webserver started at http://127.1.155.129:46109/ using document root <none> and password file <none>
I20260620 21:45:51.700105 7820 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:51.700198 7820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:51.700454 7820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:51.703437 7820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "b2a89e130a404ae68c475898d6f89967"
format_stamp: "Formatted at 2026-06-20 21:45:51 on dist-test-slave-5bvn"
server_key: "1d40f76de17ac17504b8e9cfa85a6863"
server_key_iv: "8cf7b3af7ad8788ddff4686ad84176b3"
server_key_version: "encryptionkey@0"
I20260620 21:45:51.704177 7820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "b2a89e130a404ae68c475898d6f89967"
format_stamp: "Formatted at 2026-06-20 21:45:51 on dist-test-slave-5bvn"
server_key: "1d40f76de17ac17504b8e9cfa85a6863"
server_key_iv: "8cf7b3af7ad8788ddff4686ad84176b3"
server_key_version: "encryptionkey@0"
I20260620 21:45:51.709131 7820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 21:45:51.711683 7835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:51.713294 7820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 21:45:51.713413 7820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "b2a89e130a404ae68c475898d6f89967"
format_stamp: "Formatted at 2026-06-20 21:45:51 on dist-test-slave-5bvn"
server_key: "1d40f76de17ac17504b8e9cfa85a6863"
server_key_iv: "8cf7b3af7ad8788ddff4686ad84176b3"
server_key_version: "encryptionkey@0"
I20260620 21:45:51.713544 7820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:51.742794 7820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:51.746299 7820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:51.746459 7820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:51.747017 7820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:51.747898 7820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:51.747946 7820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:51.747984 7820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:51.748000 7820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:51.756512 7820 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:41139
I20260620 21:45:51.756521 7948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:41139 every 8 connection(s)
I20260620 21:45:51.757383 7820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:45:51.757758 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 7820
I20260620 21:45:51.757918 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:45:51.758271 1646 external_mini_cluster.cc:1428] Setting key 376add47cb50eb5f2e92c3e582704249
WARNING: no policy specified for oryx/127.1.155.130@KRBTEST.COM; defaulting to no policy
Jun 20 21:45:51 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991951, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.155.130@KRBTEST.COM" created.
I20260620 21:45:51.773785 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:51.759279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:51963 (local address 127.1.155.190:36873)
0620 21:45:51.759409 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:51.759419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:51.760450 (+ 1031us) server_negotiation.cc:407] Connection header received
0620 21:45:51.762266 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:51.762271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:51.762333 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:51.762433 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:51.764353 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:51.765009 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:51.766418 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:51.766832 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:51.766951 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:51.770011 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:51.770014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:51.770016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:51.770040 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:51.771602 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:51.772211 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:51.772221 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:51.772223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:51.772302 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:51.772663 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:51.772666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:51.772668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:51.772851 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:51.772961 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:51.773493 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 21:45:51.773616 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:45:51.774572 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:51.759751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.129:51963)
0620 21:45:51.760270 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:51.760330 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:51.762067 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:51.762632 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:51.762640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:51.763231 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:51.764047 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:51.764073 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:51.765148 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:51.765153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:51.766289 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:51.766299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:51.767003 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:51.767006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:51.767155 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:51.767849 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:51.767869 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:51.769739 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:51.771770 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:51.771779 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:51.771797 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:51.772091 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:51.772397 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:51.772402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:51.772404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:51.772540 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:51.772969 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:51.772979 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:51.773214 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 21:45:51.773395 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 21:45:51.773721 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
I20260620 21:45:51.775861 7949 heartbeater.cc:347] Connected to a master server at 127.1.155.190:36873
I20260620 21:45:51.776098 7949 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:51.776572 7949 heartbeater.cc:511] Master 127.1.155.190:36873 requested a full tablet report, sending...
I20260620 21:45:51.778100 7758 ts_manager.cc:194] Registered new tserver with Master: b2a89e130a404ae68c475898d6f89967 (127.1.155.129:41139)
I20260620 21:45:51.779188 7758 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:51963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.130.keytab.
Entry for principal oryx/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.130.keytab.
I20260620 21:45:51.788971 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:36873
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.130.keytab
--principal=oryx/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:51.899232 7954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:51.899477 7954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:51.899566 7954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:51.903607 7954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:51.903661 7954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:51.903720 7954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:45:51.909224 7954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.130.keytab
--principal=oryx/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:36873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:51.910248 7954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:51.911175 7954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:51.920814 7962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:51.920883 7960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:51.921002 7954 server_base.cc:1034] running on GCE node
W20260620 21:45:51.921028 7959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:51.921417 7954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:51.922109 7954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:51.923307 7954 hybrid_clock.cc:648] HybridClock initialized: now 1781991951923275 us; error 38 us; skew 500 ppm
Jun 20 21:45:51 dist-test-slave-5bvn krb5kdc[7713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991951, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:51.926158 7954 init.cc:377] Logged in from keytab as oryx/127.1.155.130@KRBTEST.COM (short username oryx)
I20260620 21:45:51.927119 7954 webserver.cc:460] Webserver started at http://127.1.155.130:39183/ using document root <none> and password file <none>
I20260620 21:45:51.927683 7954 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:51.927767 7954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:51.927982 7954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:51.930104 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "ab51c4cca72e4d74a82e11db2a47fc40"
format_stamp: "Formatted at 2026-06-20 21:45:51 on dist-test-slave-5bvn"
server_key: "8bce0e4eb96fd31402233d01fa19741e"
server_key_iv: "a9f09b7e0bd71fc348fcf64b708fad45"
server_key_version: "encryptionkey@0"
I20260620 21:45:51.930687 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "ab51c4cca72e4d74a82e11db2a47fc40"
format_stamp: "Formatted at 2026-06-20 21:45:51 on dist-test-slave-5bvn"
server_key: "8bce0e4eb96fd31402233d01fa19741e"
server_key_iv: "a9f09b7e0bd71fc348fcf64b708fad45"
server_key_version: "encryptionkey@0"
I20260620 21:45:51.934386 7954 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:45:51.936751 7969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:51.938035 7954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:45:51.938158 7954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "ab51c4cca72e4d74a82e11db2a47fc40"
format_stamp: "Formatted at 2026-06-20 21:45:51 on dist-test-slave-5bvn"
server_key: "8bce0e4eb96fd31402233d01fa19741e"
server_key_iv: "a9f09b7e0bd71fc348fcf64b708fad45"
server_key_version: "encryptionkey@0"
I20260620 21:45:51.938267 7954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:51.964646 7954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:51.967752 7954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:51.967967 7954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:51.968535 7954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:51.969342 7954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:51.969393 7954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:51.969434 7954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:51.969449 7954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:51.978539 7954 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:36809
I20260620 21:45:51.978559 8082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:36809 every 8 connection(s)
I20260620 21:45:51.979451 7954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:45:51.985221 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 7954
I20260620 21:45:51.985369 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:45:51.985716 1646 external_mini_cluster.cc:1428] Setting key a1e424649345f93e2809172bd0335e34
Jun 20 21:45:51 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991951, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
I20260620 21:45:51.994753 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:51.981400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:34553 (local address 127.1.155.190:36873)
0620 21:45:51.981551 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:51.981555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:51.982481 (+ 926us) server_negotiation.cc:407] Connection header received
0620 21:45:51.983875 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:51.983879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:51.983942 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:51.984049 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:51.985245 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:51.985957 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:51.987317 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:51.987570 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:51.987690 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:51.990748 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:51.990751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:51.990753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:51.990776 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:51.992257 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:51.992954 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:51.992957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:51.992958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:51.993028 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:51.993500 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:51.993503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:51.993504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:51.993680 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:51.993766 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:51.994515 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 21:45:51.994635 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:45:51.996006 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:51.981750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.130:34553)
0620 21:45:51.982260 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:51.982302 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:51.983636 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:51.984216 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:51.984223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:51.984580 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:51.985077 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:51.985091 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:51.986085 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:51.986090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:51.987191 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:51.987200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:51.987723 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:51.987727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:51.987907 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:51.988606 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:51.988623 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:51.990548 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:51.992390 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:51.992398 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:51.992420 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:51.992844 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:51.993130 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:51.993136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:51.993141 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:51.993364 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:51.993807 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:51.993817 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:51.994197 (+ 380us) client_negotiation.cc:769] Sending connection context
0620 21:45:51.994454 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 21:45:51.994828 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for oryx/127.1.155.131@KRBTEST.COM; defaulting to no policy
I20260620 21:45:51.997279 8083 heartbeater.cc:347] Connected to a master server at 127.1.155.190:36873
I20260620 21:45:51.997575 8083 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:51.998121 8083 heartbeater.cc:511] Master 127.1.155.190:36873 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.155.131@KRBTEST.COM" created.
I20260620 21:45:51.999238 7758 ts_manager.cc:194] Registered new tserver with Master: ab51c4cca72e4d74a82e11db2a47fc40 (127.1.155.130:36809)
I20260620 21:45:51.999858 7758 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:34553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.131.keytab.
Entry for principal oryx/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.131.keytab.
I20260620 21:45:52.017694 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:36873
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.131.keytab
--principal=oryx/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:52.127856 8088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:52.128124 8088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:52.128165 8088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:52.132181 8088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:52.132242 8088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:52.132299 8088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:45:52.138634 8088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.131.keytab
--principal=oryx/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:36873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:52.140043 8088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:52.141402 8088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:52.151818 8093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:52.151903 8096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:52.151818 8094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:52.152019 8088 server_base.cc:1034] running on GCE node
I20260620 21:45:52.152454 8088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:52.153123 8088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:52.154286 8088 hybrid_clock.cc:648] HybridClock initialized: now 1781991952154267 us; error 31 us; skew 500 ppm
Jun 20 21:45:52 dist-test-slave-5bvn krb5kdc[7713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:52.156929 8088 init.cc:377] Logged in from keytab as oryx/127.1.155.131@KRBTEST.COM (short username oryx)
I20260620 21:45:52.157792 8088 webserver.cc:460] Webserver started at http://127.1.155.131:45425/ using document root <none> and password file <none>
I20260620 21:45:52.158341 8088 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:52.158402 8088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:52.158567 8088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:52.160490 8088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "231e4941b8d34841bd74c6b0aea79ab4"
format_stamp: "Formatted at 2026-06-20 21:45:52 on dist-test-slave-5bvn"
server_key: "92501d7d0aeaf61543869d5593af8e28"
server_key_iv: "cb8ccfe897c44e167f5eb208ae286849"
server_key_version: "encryptionkey@0"
I20260620 21:45:52.161002 8088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "231e4941b8d34841bd74c6b0aea79ab4"
format_stamp: "Formatted at 2026-06-20 21:45:52 on dist-test-slave-5bvn"
server_key: "92501d7d0aeaf61543869d5593af8e28"
server_key_iv: "cb8ccfe897c44e167f5eb208ae286849"
server_key_version: "encryptionkey@0"
I20260620 21:45:52.165195 8088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 21:45:52.167984 8103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:52.169354 8088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:45:52.169494 8088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "231e4941b8d34841bd74c6b0aea79ab4"
format_stamp: "Formatted at 2026-06-20 21:45:52 on dist-test-slave-5bvn"
server_key: "92501d7d0aeaf61543869d5593af8e28"
server_key_iv: "cb8ccfe897c44e167f5eb208ae286849"
server_key_version: "encryptionkey@0"
I20260620 21:45:52.169610 8088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:52.202983 8088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:52.206532 8088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:52.206719 8088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:52.207319 8088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:52.208123 8088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:52.208168 8088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:52.208232 8088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:52.208273 8088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:52.218024 8088 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:41081
I20260620 21:45:52.218053 8216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:41081 every 8 connection(s)
I20260620 21:45:52.218935 8088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:45:52.224411 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 8088
I20260620 21:45:52.224510 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:45:52.224789 1646 external_mini_cluster.cc:1428] Setting key b87a375720c0dc3f69acb77fb985a402
Jun 20 21:45:52 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
I20260620 21:45:52.233608 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.220903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:60591 (local address 127.1.155.190:36873)
0620 21:45:52.221069 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.221074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.222044 (+ 970us) server_negotiation.cc:407] Connection header received
0620 21:45:52.223463 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.223465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.223515 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.223589 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.224931 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.225585 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.227377 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.227587 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.227666 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.230368 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.230372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.230377 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.230406 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.231723 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.232232 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.232235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.232236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.232306 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.232653 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.232655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.232656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.232821 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.232904 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.233393 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.233492 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 21:45:52.234481 8219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.221309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.131:60591)
0620 21:45:52.221822 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.221943 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.223256 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.223779 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.223786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.224157 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:52.224742 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.224760 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.225763 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.225770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.227224 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.227234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.227737 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.227741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.227893 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.228617 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:52.228637 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:52.230197 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:52.231835 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.231840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.231852 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.232134 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.232405 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.232407 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.232409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.232556 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.232926 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:52.232932 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:52.233166 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.233331 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.233589 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":99,"threads_started":1}
I20260620 21:45:52.235584 8217 heartbeater.cc:347] Connected to a master server at 127.1.155.190:36873
I20260620 21:45:52.235816 8217 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:52.236274 8217 heartbeater.cc:511] Master 127.1.155.190:36873 requested a full tablet report, sending...
I20260620 21:45:52.237231 7758 ts_manager.cc:194] Registered new tserver with Master: 231e4941b8d34841bd74c6b0aea79ab4 (127.1.155.131:41081)
I20260620 21:45:52.237802 7758 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:60591
I20260620 21:45:52.238958 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.155.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.155.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.155.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.189.keytab.
Entry for principal oryx/127.1.155.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.189.keytab.
I20260620 21:45:52.266125 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.189:38729
--webserver_interface=127.1.155.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.189.keytab
--principal=oryx/127.1.155.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:36873,127.1.155.189:38729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:52.373822 8222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:52.374162 8222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:52.374253 8222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:52.378425 8222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:52.378504 8222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:52.378540 8222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:52.378597 8222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:52.378633 8222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:45:52.384078 8222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:36873,127.1.155.189:38729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.189.keytab
--principal=oryx/127.1.155.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.189:38729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.155.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
W20260620 21:45:52.384318 8222 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.155.190:36873,127.1.155.189:38729'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 21:45:52.385215 8222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:52.386263 8222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:52.394246 8230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:52.394263 8227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:52.394234 8228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:52.394533 8222 server_base.cc:1034] running on GCE node
I20260620 21:45:52.395061 8222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:52.396160 8222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:52.397352 8222 hybrid_clock.cc:648] HybridClock initialized: now 1781991952397325 us; error 46 us; skew 500 ppm
Jun 20 21:45:52 dist-test-slave-5bvn krb5kdc[7713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:52.400043 8222 init.cc:377] Logged in from keytab as oryx/127.1.155.189@KRBTEST.COM (short username oryx)
I20260620 21:45:52.400993 8222 webserver.cc:460] Webserver started at http://127.1.155.189:44501/ using document root <none> and password file <none>
I20260620 21:45:52.401517 8222 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:52.401574 8222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:52.401731 8222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:52.403770 8222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/data/instance:
uuid: "0c67cbaf33094ee3a55bb073c12e4749"
format_stamp: "Formatted at 2026-06-20 21:45:52 on dist-test-slave-5bvn"
server_key: "bd202b2647aee6b542bc5e9d3e438313"
server_key_iv: "61659b10cdf46fac81923d2419a9f786"
server_key_version: "encryptionkey@0"
I20260620 21:45:52.404264 8222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/wal/instance:
uuid: "0c67cbaf33094ee3a55bb073c12e4749"
format_stamp: "Formatted at 2026-06-20 21:45:52 on dist-test-slave-5bvn"
server_key: "bd202b2647aee6b542bc5e9d3e438313"
server_key_iv: "61659b10cdf46fac81923d2419a9f786"
server_key_version: "encryptionkey@0"
I20260620 21:45:52.407852 8222 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:45:52.410137 8237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:52.411460 8222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:45:52.411587 8222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/wal
uuid: "0c67cbaf33094ee3a55bb073c12e4749"
format_stamp: "Formatted at 2026-06-20 21:45:52 on dist-test-slave-5bvn"
server_key: "bd202b2647aee6b542bc5e9d3e438313"
server_key_iv: "61659b10cdf46fac81923d2419a9f786"
server_key_version: "encryptionkey@0"
I20260620 21:45:52.411687 8222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:52.433111 8222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:52.436136 8222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:52.436354 8222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:52.443492 8222 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.189:38729
I20260620 21:45:52.443531 8289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.189:38729 every 8 connection(s)
I20260620 21:45:52.444555 8222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/data/info.pb
I20260620 21:45:52.447410 8290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:52.451061 8290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:45:52.451988 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 8222
Jun 20 21:45:52 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.189@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
Jun 20 21:45:52 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.155.189@KRBTEST.COM
I20260620 21:45:52.464609 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.453001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.1.155.190:36873)
0620 21:45:52.453124 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.453128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.454199 (+ 1071us) server_negotiation.cc:407] Connection header received
0620 21:45:52.455034 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.455037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.455093 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.455213 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.456000 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.456529 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.457527 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.457742 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.457913 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.460475 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.460479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.460481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.460512 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.462170 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.462757 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.462760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.462761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.462853 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.463205 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.463209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.463211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.463521 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.463620 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.464248 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.464449 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:45:52.465502 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.453381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.1.155.189:38729)
0620 21:45:52.453997 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.454015 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.454051 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:45:52.454800 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.454822 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.455053 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.455230 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.456019 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.456834 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.457622 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.458046 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.458159 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.460898 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.460905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.460919 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.460958 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.462876 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.463388 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.463393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.463395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.463475 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.463844 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.463847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.463848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.464060 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.464231 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.464526 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.464802 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"spinlock_wait_cycles":28032,"thread_start_us":152,"threads_started":1}
I20260620 21:45:52.465507 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.453381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.0.0.1:46250)
0620 21:45:52.454052 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.454100 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.454800 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.455266 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.455272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.455617 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:52.455858 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.455871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.456662 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.456665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.457400 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.457408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.457948 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.457951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.458103 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.458689 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:52.458704 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:52.460291 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:52.462277 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.462285 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.462304 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.462647 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.462948 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.462952 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.462957 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.463107 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.463624 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:52.463631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:52.463940 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.464186 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.464580 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":125,"threads_started":1}
I20260620 21:45:52.466902 8290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:45:52 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.189@KRBTEST.COM for oryx/127.1.155.189@KRBTEST.COM
I20260620 21:45:52.474942 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.467412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38729 (local address 127.0.0.1:48358)
0620 21:45:52.467535 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.467552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.467620 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.467840 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.467842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.467969 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:52.468049 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.468054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.468730 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.468733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.469706 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.469713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.470237 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.470240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.470324 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.470818 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:52.470829 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:52.471986 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:52.473612 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.473615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.473617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.473791 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.474107 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.474110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.474111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.474194 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.474573 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:52.474576 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:52.474651 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.474741 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.474834 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 21:45:52.475071 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.467496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.1.155.189:38729)
0620 21:45:52.467614 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.467618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.467632 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:45:52.467674 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.467680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.467732 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.467809 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.468148 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.468637 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.469876 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.470119 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.470238 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.472118 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.472121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.472124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.472146 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.473498 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.473960 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.473962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.473963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.474022 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.474321 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.474323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.474324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.474482 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.474557 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.474811 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.474897 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:45:52.477537 8290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: Bootstrap starting.
I20260620 21:45:52.479748 8290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:52.480499 8290 log.cc:824] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:52.482409 8290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: No bootstrap required, opened a new log
I20260620 21:45:52.485266 8290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } }
I20260620 21:45:52.485451 8290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:52.485509 8290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c67cbaf33094ee3a55bb073c12e4749, State: Initialized, Role: FOLLOWER
I20260620 21:45:52.485965 8290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } }
I20260620 21:45:52.486438 8296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } } }
I20260620 21:45:52.486598 8296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:45:52.487136 8290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 21:45:52.491714 8307 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:45:52.491811 8307 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:45:52.494117 8222 master_runner.cc:428] Detected that this master 0c67cbaf33094ee3a55bb073c12e4749 is joining an existing cluster
I20260620 21:45:52.494184 8222 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.155.189:38729
I20260620 21:45:52.504930 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.496125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.1.155.190:36873)
0620 21:45:52.496232 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.496237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.496521 (+ 284us) server_negotiation.cc:407] Connection header received
0620 21:45:52.496705 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.496708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.496774 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.496825 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.497532 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.498311 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.499149 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.499378 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.499489 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.501494 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.501497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.501500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.501524 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.503249 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.503696 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.503699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.503700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.503771 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.504064 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.504069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.504071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.504278 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.504367 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.504617 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.504754 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 21:45:52.504953 8313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.496065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.0.0.1:46264)
0620 21:45:52.496439 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.496455 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.496593 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.496965 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.496968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.497264 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:52.497428 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.497435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.498393 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.498399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.499030 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.499037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.499610 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.499613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.499742 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.500748 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:52.500767 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:52.501335 (+ 568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:52.503367 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.503370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.503373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.503611 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.503873 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.503876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.503878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.503975 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.504356 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:52.504359 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:52.504457 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.504564 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.504773 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":63,"spinlock_wait_cycles":11008,"thread_start_us":100,"threads_started":1}
I20260620 21:45:52.505206 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.496125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.1.155.189:38729)
0620 21:45:52.496232 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.496237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.496651 (+ 414us) server_negotiation.cc:407] Connection header received
0620 21:45:52.496815 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.496817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.496852 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.496938 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.497460 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.498238 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.499458 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.499787 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.499897 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.501599 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.501603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.501605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.501628 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.503411 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.504063 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.504066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.504068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.504134 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.504380 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.504383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.504385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.504562 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.504624 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.504958 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.505057 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":39,"spinlock_wait_cycles":768}
I20260620 21:45:52.505585 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.496316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38729 (local address 127.0.0.1:48368)
0620 21:45:52.496560 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.496579 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.496698 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.496951 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.496953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.497221 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:52.497350 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.497356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.498359 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.498363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.499337 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.499347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.499909 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.499913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.500343 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.500956 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:52.500971 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:52.501466 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:52.503736 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.503740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.503744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.503968 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.504224 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.504226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.504228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.504297 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.504679 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:52.504681 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:52.504815 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.505321 (+ 506us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.505433 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":358,"thread_start_us":100,"threads_started":1}
Jun 20 21:45:52 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991951, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.190@KRBTEST.COM for oryx/127.1.155.189@KRBTEST.COM
I20260620 21:45:52.518973 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.509254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38729 (local address 127.0.0.1:48380)
0620 21:45:52.509655 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.509721 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.509884 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.510165 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.510171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.510594 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:52.510726 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.510733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.511388 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.511392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.512368 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.512378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.512772 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.512775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.512896 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.513431 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:52.513445 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:52.515017 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:52.516903 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.516914 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.516920 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.517187 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.517550 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.517553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.517556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.517659 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.518208 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:52.518216 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:52.518377 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.518550 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.518774 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":116,"threads_started":1}
I20260620 21:45:52.519155 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.509294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.1.155.189:38729)
0620 21:45:52.509422 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.509426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.509803 (+ 377us) server_negotiation.cc:407] Connection header received
0620 21:45:52.510002 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.510004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.510045 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.510108 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.510841 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.511288 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.512482 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.512671 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.512745 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.515155 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.515158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.515161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.515183 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.516779 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.517316 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.517323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.517325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.517434 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.517760 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.517763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.517764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.518073 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.518207 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.518614 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.519009 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:45:52.520362 7758 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.155.189:38729: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749"
member_type: NON_VOTER
last_known_addr {
host: "127.1.155.189"
port: 38729
}
attrs {
promote: true
}
}
dest_uuid: "32c4b8ae8381486abded8f30d55598ee"
cas_config_opid_index: -1
I20260620 21:45:52.521147 7758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } }
I20260620 21:45:52.523173 8318 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0c67cbaf33094ee3a55bb073c12e4749 (127.1.155.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } }
I20260620 21:45:52.523788 8264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:52.524991 8318 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.155.189:38729
I20260620 21:45:52.525203 8264 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 FOLLOWER]: Refusing update from remote peer 32c4b8ae8381486abded8f30d55598ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 21:45:52.525409 8222 master.cc:537] Master@127.1.155.189:38729 shutting down...
I20260620 21:45:52.527283 8318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } }
I20260620 21:45:52.527678 8318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: This master's current role is: LEADER
WARNING: no policy specified for oryx/127.1.155.188@KRBTEST.COM; defaulting to no policy
I20260620 21:45:52.527904 8318 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 21:45:52.530236 7745 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.155.188@KRBTEST.COM" created.
W20260620 21:45:52.531116 7745 consensus_peers.cc:487] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee -> Peer 0c67cbaf33094ee3a55bb073c12e4749 (127.1.155.189:38729): Couldn't send request to peer 0c67cbaf33094ee3a55bb073c12e4749. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 21:45:52.532297 8222 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:52.532577 8222 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:52.532676 8222 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.155.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.188.keytab.
Entry for principal oryx/127.1.155.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.188.keytab.
I20260620 21:45:52.546921 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.188:40219
--webserver_interface=127.1.155.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.188.keytab
--principal=oryx/127.1.155.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:36873,127.1.155.189:38729,127.1.155.188:40219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:45:52.547608 8222 master.cc:552] Master@127.1.155.189:38729 shutdown complete.
I20260620 21:45:52.547662 8222 master_runner.cc:305] Clearing existing system tablet
I20260620 21:45:52.547909 8222 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:52.547960 8222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:52.549337 8222 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:45:52.550307 8326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:52.550554 8222 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:52.550674 8222 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 21:45:52.550715 8222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/wal
uuid: "0c67cbaf33094ee3a55bb073c12e4749"
format_stamp: "Formatted at 2026-06-20 21:45:52 on dist-test-slave-5bvn"
server_key: "bd202b2647aee6b542bc5e9d3e438313"
server_key_iv: "61659b10cdf46fac81923d2419a9f786"
server_key_version: "encryptionkey@0"
I20260620 21:45:52.552783 8222 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:52.555128 8222 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 21:45:52.555218 8222 log.cc:1196] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 21:45:52.555462 8222 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: Deleting consensus metadata
I20260620 21:45:52.555639 8222 master_runner.cc:315] Copying system tablet from 127.1.155.190:36873
I20260620 21:45:52.556867 8222 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: tablet copy: Beginning tablet copy session from remote peer at address 127.1.155.190:36873
I20260620 21:45:52.566135 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.557551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46276 (local address 127.1.155.190:36873)
0620 21:45:52.557674 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.557692 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.557874 (+ 182us) server_negotiation.cc:407] Connection header received
0620 21:45:52.558001 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.558005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.558056 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.558150 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.558787 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.559628 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.560810 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.561167 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.561270 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.562586 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.562590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.562593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.562619 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.564416 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.564875 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.564878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.564880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.564965 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.565251 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.565253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.565255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.565461 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.565559 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.565865 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.565980 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:45:52.566139 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.557450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.0.0.1:46276)
0620 21:45:52.557756 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.557772 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.557913 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.558275 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.558278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.558502 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:52.558628 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.558636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.559749 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.559753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.560653 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.560694 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.561302 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.561306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.561402 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.562018 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:52.562033 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:52.562440 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:52.564513 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.564517 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.564519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.564770 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.565067 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.565070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.565072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.565175 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.565552 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:52.565560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:52.565656 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.565769 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.565928 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
I20260620 21:45:52.567030 7778 tablet_copy_service.cc:140] P 32c4b8ae8381486abded8f30d55598ee: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0c67cbaf33094ee3a55bb073c12e4749 ({username='oryx', principal='oryx/127.1.155.189@KRBTEST.COM'} at 127.0.0.1:46276)
I20260620 21:45:52.567214 7778 tablet_copy_service.cc:161] P 32c4b8ae8381486abded8f30d55598ee: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0c67cbaf33094ee3a55bb073c12e4749 at {username='oryx', principal='oryx/127.1.155.189@KRBTEST.COM'} at 127.0.0.1:46276: session id = 0c67cbaf33094ee3a55bb073c12e4749-00000000000000000000000000000000
I20260620 21:45:52.569031 7778 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 21:45:52.570842 8222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 21:45:52.573276 8222 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: tablet copy: Starting download of 0 data blocks...
I20260620 21:45:52.573527 8222 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: tablet copy: Starting download of 1 WAL segments...
I20260620 21:45:52.575316 8222 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 21:45:52.577140 7778 tablet_copy_service.cc:342] P 32c4b8ae8381486abded8f30d55598ee: Request end of tablet copy session 0c67cbaf33094ee3a55bb073c12e4749-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.155.189@KRBTEST.COM'} at 127.0.0.1:46276
I20260620 21:45:52.577263 7778 tablet_copy_service.cc:433] P 32c4b8ae8381486abded8f30d55598ee: ending tablet copy session 0c67cbaf33094ee3a55bb073c12e4749-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0c67cbaf33094ee3a55bb073c12e4749
I20260620 21:45:52.584267 8222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:52.585987 8333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:52.586136 8334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:52.586414 8336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:52.586963 8222 server_base.cc:1034] running on GCE node
I20260620 21:45:52.587179 8222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:52.587622 8222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:52.588824 8222 hybrid_clock.cc:648] HybridClock initialized: now 1781991952588815 us; error 31 us; skew 500 ppm
Jun 20 21:45:52 dist-test-slave-5bvn krb5kdc[7713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:52.590534 8222 init.cc:377] Logged in from keytab as oryx/127.1.155.189@KRBTEST.COM (short username oryx)
I20260620 21:45:52.591030 8222 webserver.cc:460] Webserver started at http://127.1.155.189:39343/ using document root <none> and password file <none>
I20260620 21:45:52.591274 8222 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:52.591357 8222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:52.592859 8222 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:52.593928 8343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:52.594223 8222 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:52.594292 8222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/wal
uuid: "0c67cbaf33094ee3a55bb073c12e4749"
format_stamp: "Formatted at 2026-06-20 21:45:52 on dist-test-slave-5bvn"
server_key: "bd202b2647aee6b542bc5e9d3e438313"
server_key_iv: "61659b10cdf46fac81923d2419a9f786"
server_key_version: "encryptionkey@0"
I20260620 21:45:52.594362 8222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:52.612355 8222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:52.614630 8222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:52.619503 8222 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.189:38729
I20260620 21:45:52.619539 8395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.189:38729 every 8 connection(s)
I20260620 21:45:52.620223 8222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-1/data/info.pb
I20260620 21:45:52.622191 8396 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:45:52.623230 8396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: Bootstrap starting.
I20260620 21:45:52.633947 8396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:45:52.634303 8396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: Bootstrap complete.
I20260620 21:45:52.634924 8396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } }
I20260620 21:45:52.635082 8396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0c67cbaf33094ee3a55bb073c12e4749, State: Initialized, Role: LEARNER
I20260620 21:45:52.635202 8396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } }
I20260620 21:45:52.635571 8399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } }
I20260620 21:45:52.635703 8396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:52.635744 8399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: This master's current role is: LEARNER
I20260620 21:45:52.639619 8410 catalog_manager.cc:1259] Loaded cluster ID: 4a3ea318e1ba4c3598f9fdf369f79031
I20260620 21:45:52.639688 8410 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: loading cluster ID for follower catalog manager: success
I20260620 21:45:52.641870 8410 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: acquiring CA information for follower catalog manager: success
I20260620 21:45:52.643340 8410 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 21:45:52.680655 8324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:52.680935 8324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:52.680977 8324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:52.684912 8324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:52.684968 8324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:52.684988 8324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:52.685005 8324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:52.685020 8324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:45:52.690280 8324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:36873,127.1.155.189:38729,127.1.155.188:40219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.188.keytab
--principal=oryx/127.1.155.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.188:40219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.155.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:52.691278 8324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:52.692224 8324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:52.700599 8418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:52.700619 8415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:52.700613 8416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:52.700899 8324 server_base.cc:1034] running on GCE node
I20260620 21:45:52.701437 8324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:52.702608 8324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:52.703796 8324 hybrid_clock.cc:648] HybridClock initialized: now 1781991952703793 us; error 35 us; skew 500 ppm
Jun 20 21:45:52 dist-test-slave-5bvn krb5kdc[7713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:52.706423 8324 init.cc:377] Logged in from keytab as oryx/127.1.155.188@KRBTEST.COM (short username oryx)
I20260620 21:45:52.707257 8324 webserver.cc:460] Webserver started at http://127.1.155.188:33711/ using document root <none> and password file <none>
I20260620 21:45:52.707795 8324 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:52.707893 8324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:52.708118 8324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:52.710131 8324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/data/instance:
uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db"
format_stamp: "Formatted at 2026-06-20 21:45:52 on dist-test-slave-5bvn"
server_key: "f411a987e603051a4277fcda78a746b7"
server_key_iv: "ac9a2e171cc17d39900395d680c57027"
server_key_version: "encryptionkey@0"
I20260620 21:45:52.710693 8324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/wal/instance:
uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db"
format_stamp: "Formatted at 2026-06-20 21:45:52 on dist-test-slave-5bvn"
server_key: "f411a987e603051a4277fcda78a746b7"
server_key_iv: "ac9a2e171cc17d39900395d680c57027"
server_key_version: "encryptionkey@0"
I20260620 21:45:52.714373 8324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:45:52.716691 8425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:52.718026 8324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:45:52.718155 8324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/wal
uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db"
format_stamp: "Formatted at 2026-06-20 21:45:52 on dist-test-slave-5bvn"
server_key: "f411a987e603051a4277fcda78a746b7"
server_key_iv: "ac9a2e171cc17d39900395d680c57027"
server_key_version: "encryptionkey@0"
I20260620 21:45:52.718266 8324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:52.747408 8324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:52.750341 8324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:52.750540 8324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:52.758128 8324 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.188:40219
I20260620 21:45:52.758481 8477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.188:40219 every 8 connection(s)
I20260620 21:45:52.759277 8324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/data/info.pb
I20260620 21:45:52.761998 8478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:52.764295 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 8324
I20260620 21:45:52.765952 8478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:45:52 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.188@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
Jun 20 21:45:52 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.155.188@KRBTEST.COM
I20260620 21:45:52.778993 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.767956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.1.155.190:36873)
0620 21:45:52.768082 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.768086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.768440 (+ 354us) server_negotiation.cc:407] Connection header received
0620 21:45:52.768597 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.768605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.768645 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.768733 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.769440 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.770215 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.771565 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.771888 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.771981 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.775005 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.775008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.775010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.775033 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.776752 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.777333 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.777338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.777340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.777428 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.777791 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.777794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.777796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.778080 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.778174 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.778681 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.778828 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:45:52.779783 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.765761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46262 (local address 127.1.155.188:40219)
0620 21:45:52.766218 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.766238 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.766280 (+ 42us) server_negotiation.cc:407] Connection header received
0620 21:45:52.767078 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.767099 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.767484 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.767744 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.768829 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.770132 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.771313 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.771807 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.771976 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.775448 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.775461 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.775488 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.775514 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.777332 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.777801 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.777807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.777809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.777919 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.778201 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.778204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.778206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.778465 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.778636 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.778873 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.778996 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":119,"threads_started":1}
I20260620 21:45:52.779788 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.767834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.0.0.1:46280)
0620 21:45:52.768310 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.768352 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.768494 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.768781 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.768786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.769107 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:52.769251 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.769260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.770384 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.770387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.771441 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.771449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.772026 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.772030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.772145 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.772849 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:52.772872 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:52.774809 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:52.776875 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.776882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.776890 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.777238 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.777520 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.777525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.777530 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.777698 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.778179 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:52.778188 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:52.778431 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.778608 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.778949 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":137,"threads_started":1}
I20260620 21:45:52.781018 8478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:45:52.781762 7949 heartbeater.cc:503] Master 127.1.155.190:36873 was elected leader, sending a full tablet report...
Jun 20 21:45:52 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.188@KRBTEST.COM for oryx/127.1.155.189@KRBTEST.COM
I20260620 21:45:52.789252 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.781499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38729 (local address 127.0.0.1:48382)
0620 21:45:52.781803 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.781820 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.781953 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.782277 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.782280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.782446 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:52.782528 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.782533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.783450 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.783453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.783992 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.783999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.784454 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.784458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.784548 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.784958 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:52.784969 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:52.786310 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:52.787886 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.787888 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.787890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.788082 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.788472 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.788474 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.788475 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.788547 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.788877 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:52.788879 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:52.788950 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.789044 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.789139 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234}
I20260620 21:45:52.789417 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.781582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.1.155.189:38729)
0620 21:45:52.782039 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.782042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.782055 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:45:52.782119 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.782121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.782160 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.782265 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.782647 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.783336 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.784114 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.784349 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.784459 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.786437 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.786440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.786442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.786464 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.787788 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.788279 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.788282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.788284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.788379 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.788626 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.788628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.788630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.788794 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.788873 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.789100 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.789200 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":82,"threads_started":1}
I20260620 21:45:52.790113 8478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:45:52 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.188@KRBTEST.COM for oryx/127.1.155.188@KRBTEST.COM
I20260620 21:45:52.797259 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.790633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46266 (local address 127.1.155.188:40219)
0620 21:45:52.790715 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.790719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.790731 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:45:52.790815 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.790817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.790856 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.790951 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.791312 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.791811 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.792476 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.792689 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.792762 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.794523 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.794525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.794527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.794546 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.795956 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.796351 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.796354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.796355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.796419 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.796649 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.796651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.796652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.796810 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.796881 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.797080 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.797160 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 21:45:52.797267 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.790534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:40219 (local address 127.0.0.1:46266)
0620 21:45:52.790646 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.790661 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.790730 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.790974 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.790977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.791099 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:52.791164 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.791168 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.791900 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.791902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.792383 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.792390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.792777 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.792779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.792832 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.793201 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:52.793211 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:52.794408 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:52.796061 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.796064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.796065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.796263 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.796499 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.796501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.796502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.796573 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.796886 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:52.796887 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:52.796954 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.797030 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.797116 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 21:45:52.800871 8478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: Bootstrap starting.
I20260620 21:45:52.803148 8478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:52.803948 8478 log.cc:824] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:52.806087 8478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: No bootstrap required, opened a new log
I20260620 21:45:52.808981 8478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } }
I20260620 21:45:52.809193 8478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:52.809288 8478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec4a9136bbd4d87ac89b1d9a9b104db, State: Initialized, Role: FOLLOWER
I20260620 21:45:52.809700 8478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } }
I20260620 21:45:52.810206 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } } }
I20260620 21:45:52.810329 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:45:52.810921 8478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: configured and running, proceeding with master startup.
W20260620 21:45:52.815855 8495 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:45:52.815950 8495 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:45:52.819183 8324 master_runner.cc:428] Detected that this master 0ec4a9136bbd4d87ac89b1d9a9b104db is joining an existing cluster
I20260620 21:45:52.819267 8324 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.155.188:40219
I20260620 21:45:52.821346 8484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:52.821506 8484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } }
I20260620 21:45:52.822510 8484 leader_election.cc:288] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32c4b8ae8381486abded8f30d55598ee (127.1.155.190:36873), 0c67cbaf33094ee3a55bb073c12e4749 (127.1.155.189:38729)
I20260620 21:45:52.822877 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c67cbaf33094ee3a55bb073c12e4749" is_pre_election: true
I20260620 21:45:52.823050 8370 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 LEARNER]: Handling vote request from an unknown peer 0ec4a9136bbd4d87ac89b1d9a9b104db
I20260620 21:45:52.823024 7768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32c4b8ae8381486abded8f30d55598ee" is_pre_election: true
I20260620 21:45:52.823177 8370 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 0ec4a9136bbd4d87ac89b1d9a9b104db for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260620 21:45:52.823633 8428 leader_election.cc:302] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ec4a9136bbd4d87ac89b1d9a9b104db; no voters: 0c67cbaf33094ee3a55bb073c12e4749, 32c4b8ae8381486abded8f30d55598ee
I20260620 21:45:52.824105 8484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:52.825152 8484 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 21:45:52.831429 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.822305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.1.155.190:36873)
0620 21:45:52.822614 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.822620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.822634 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:45:52.822677 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.822682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.822734 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.822811 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.824169 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.825003 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.826393 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.826607 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.826724 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.827875 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.827879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.827883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.827910 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.829540 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.830158 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.830162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.830164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.830253 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.830558 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.830561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.830563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.830759 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.830836 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.831138 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.831254 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260620 21:45:52.831934 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.821294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.0.0.1:46286)
0620 21:45:52.822137 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.822151 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.822237 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.823659 (+ 1422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.823662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.823894 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:52.824031 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.824039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.825141 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.825144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.826102 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.826109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.826722 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.826726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.826814 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.827350 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:52.827363 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:52.827706 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:52.829649 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.829654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.829659 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.830048 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.830372 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.830375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.830377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.830462 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.830889 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:52.830892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:52.830974 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.831629 (+ 655us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.831765 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"spinlock_wait_cycles":3968,"thread_start_us":103,"threads_started":1}
I20260620 21:45:52.833655 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.821775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38729 (local address 127.0.0.1:48384)
0620 21:45:52.825994 (+ 4219us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.826007 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.826105 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.826360 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.826364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.826563 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:52.826672 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.826679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.827737 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.827740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.828610 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.828619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.829142 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.829145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.829235 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.829693 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:52.829710 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:52.830207 (+ 497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:52.832070 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.832073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.832075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.832335 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.832635 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.832637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.832638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.832714 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.833149 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:52.833152 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:52.833234 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.833344 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.833479 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4156,"spinlock_wait_cycles":1408,"thread_start_us":152,"threads_started":1}
I20260620 21:45:52.833688 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.821549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48384 (local address 127.1.155.189:38729)
0620 21:45:52.821667 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.821672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.826093 (+ 4421us) server_negotiation.cc:407] Connection header received
0620 21:45:52.826189 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.826191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.826232 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.826301 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.826811 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.827597 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.828719 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.829018 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.829145 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.830336 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.830339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.830342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.830366 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.831929 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.832445 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.832448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.832450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.832531 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.832806 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.832811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.832812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.833013 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.833109 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.833413 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.833534 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 21:45:52.834556 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 7820
I20260620 21:45:52.841464 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:41139
--local_ip_for_outbound_sockets=127.1.155.129
--tserver_master_addrs=127.1.155.190:36873
--webserver_port=46109
--webserver_interface=127.1.155.129
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.129.keytab
--principal=oryx/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.155.190:36873,127.1.155.189:38729
--tserver_master_addrs=127.1.155.190:36873,127.1.155.189:38729,127.1.155.188:40219 with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:45:52 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991951, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.190@KRBTEST.COM for oryx/127.1.155.188@KRBTEST.COM
I20260620 21:45:52.846252 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.836693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:40219 (local address 127.0.0.1:46274)
0620 21:45:52.836810 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.836827 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.836905 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.837470 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.837473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.837661 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:52.837786 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.837793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.838994 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.838997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.839849 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.839859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.840479 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.840483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.840575 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.841100 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:52.841112 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:52.842605 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:52.844583 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.844586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.844588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.844843 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.845145 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.845152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.845157 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.845258 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.845653 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:52.845656 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:52.845750 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.845888 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.846030 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 21:45:52.846808 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.836929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46274 (local address 127.1.155.188:40219)
0620 21:45:52.837201 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.837209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.837222 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:45:52.837290 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.837293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.837342 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.837950 (+ 608us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.838050 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.838867 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.839993 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.840360 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.840783 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.842758 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.842762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.842765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.842788 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.844477 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.844936 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.844939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.844941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.845023 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.845384 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.845387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.845389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.845558 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.845664 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.845932 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.846038 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":107,"threads_started":1}
I20260620 21:45:52.847792 7758 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.155.188:40219: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db"
member_type: NON_VOTER
last_known_addr {
host: "127.1.155.188"
port: 40219
}
attrs {
promote: true
}
}
dest_uuid: "32c4b8ae8381486abded8f30d55598ee"
cas_config_opid_index: 5
I20260620 21:45:52.848299 7758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } }
I20260620 21:45:52.849407 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.822175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.1.155.188:40219)
0620 21:45:52.822507 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.822511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.823710 (+ 1199us) server_negotiation.cc:407] Connection header received
0620 21:45:52.823954 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.823956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.823992 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.824034 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.842378 (+ 18344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.843093 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.844325 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.844652 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.844757 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.845858 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.845861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.845864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.845884 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.847518 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.848199 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.848202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.848204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.848282 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.848625 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.848628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.848630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.848817 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.848882 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.849162 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.849271 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260620 21:45:52.849294 8319 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0ec4a9136bbd4d87ac89b1d9a9b104db (127.1.155.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } } }
I20260620 21:45:52.849962 8452 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 FOLLOWER]: Refusing update from remote peer 32c4b8ae8381486abded8f30d55598ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 21:45:52.850311 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.821716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:40219 (local address 127.0.0.1:46270)
0620 21:45:52.823633 (+ 1917us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.823648 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.823856 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.824157 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.824160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.842071 (+ 17911us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:52.842196 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.842203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.843254 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.843260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.844197 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.844207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.844772 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.844776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.844885 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.845302 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:52.845315 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:52.845704 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:52.847744 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.847747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.847750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.848099 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.848371 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.848373 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.848375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.848535 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.848937 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:52.848940 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:52.849020 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.849727 (+ 707us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.849876 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1854,"thread_start_us":132,"threads_started":1}
I20260620 21:45:52.850908 8319 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.155.188:40219
I20260620 21:45:52.851078 8319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:52.851785 8319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } } }
I20260620 21:45:52.851902 8319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:52.852383 8324 master.cc:537] Master@127.1.155.188:40219 shutting down...
I20260620 21:45:52.855316 8452 raft_consensus.cc:858] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } }
W20260620 21:45:52.855436 8452 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } }
I20260620 21:45:52.856977 8506 mvcc.cc:205] Tried to move back new op lower bound from 7299039033233461248 to 7299039033191297024. Current Snapshot: MvccSnapshot[applied={T|T < 7299039033233461248}]
I20260620 21:45:52.857740 8452 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0c67cbaf33094ee3a55bb073c12e4749 (127.1.155.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } }
I20260620 21:45:52.859802 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.852431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38729 (local address 127.0.0.1:48394)
0620 21:45:52.852995 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.853010 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.853085 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.853475 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.853478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.853494 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:52.853571 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.853576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.855768 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.855771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.856917 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.856928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.859321 (+ 2393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.859328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.859435 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.859484 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.859567 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.859659 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499}
I20260620 21:45:52.860095 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32c4b8ae8381486abded8f30d55598ee. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } }
I20260620 21:45:52.860189 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 21:45:52.860189 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.852734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.1.155.189:38729)
0620 21:45:52.852909 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.852913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.853163 (+ 250us) server_negotiation.cc:407] Connection header received
0620 21:45:52.853214 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.853217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.853272 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.853365 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:52.853683 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.854447 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.857703 (+ 3256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.858958 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.859064 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.859235 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.859987 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.860071 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 21:45:52.860280 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } }
I20260620 21:45:52.860338 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 21:45:52.860392 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } }
I20260620 21:45:52.860443 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 21:45:52.860651 8370 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 LEARNER]: Refusing update from remote peer 32c4b8ae8381486abded8f30d55598ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 21:45:52.861145 8318 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 21:45:52.863112 8370 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0ec4a9136bbd4d87ac89b1d9a9b104db (127.1.155.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } } }
I20260620 21:45:52.865828 8399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32c4b8ae8381486abded8f30d55598ee. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } } }
I20260620 21:45:52.865995 8399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: This master's current role is: LEARNER
I20260620 21:45:52.865968 8514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } } }
I20260620 21:45:52.866087 8514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: This master's current role is: LEARNER
I20260620 21:45:52.866508 8324 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 21:45:52.866606 8320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: NON_VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: true } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } } }
I20260620 21:45:52.866712 8320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:52.866794 8324 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 21:45:52.866875 8324 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: stopping tablet replica
W20260620 21:45:52.867770 7743 consensus_peers.cc:487] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee -> Peer 0ec4a9136bbd4d87ac89b1d9a9b104db (127.1.155.188:40219): Couldn't send request to peer 0ec4a9136bbd4d87ac89b1d9a9b104db. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 21:45:52.869115 8319 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee: attempting to promote NON_VOTER 0c67cbaf33094ee3a55bb073c12e4749 to VOTER
I20260620 21:45:52.869648 8319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } }
I20260620 21:45:52.870998 8370 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 LEARNER]: Refusing update from remote peer 32c4b8ae8381486abded8f30d55598ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 21:45:52.871309 8320 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 21:45:52.871411 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.871004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:40219 (local address 127.0.0.1:46278)
0620 21:45:52.871183 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.871282 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:40219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 21:45:52.872084 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.871838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:40219 (local address 127.0.0.1:46294)
0620 21:45:52.871975 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.872024 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:40219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 21:45:52.872181 7743 consensus_peers.cc:487] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee -> Peer 0ec4a9136bbd4d87ac89b1d9a9b104db (127.1.155.188:40219): Couldn't send request to peer 0ec4a9136bbd4d87ac89b1d9a9b104db. Status: Network error: Client connection negotiation failed: client connection to 127.1.155.188:40219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 21:45:52.873143 8319 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0c67cbaf33094ee3a55bb073c12e4749 (127.1.155.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } } }
I20260620 21:45:52.873548 8370 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0c67cbaf33094ee3a55bb073c12e4749 (127.1.155.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } } }
I20260620 21:45:52.874670 8319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } } }
I20260620 21:45:52.874769 8319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:52.874917 8318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } } }
I20260620 21:45:52.875030 8318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:52.875178 8514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } } }
I20260620 21:45:52.875270 8514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:45:52.884193 8324 master.cc:552] Master@127.1.155.188:40219 shutdown complete.
I20260620 21:45:52.884268 8324 master_runner.cc:305] Clearing existing system tablet
I20260620 21:45:52.884502 8324 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:52.884569 8324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:52.885717 8324 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:45:52.886613 8516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:52.886840 8324 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:52.886986 8324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:45:52.887023 8324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/wal
uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db"
format_stamp: "Formatted at 2026-06-20 21:45:52 on dist-test-slave-5bvn"
server_key: "f411a987e603051a4277fcda78a746b7"
server_key_iv: "ac9a2e171cc17d39900395d680c57027"
server_key_version: "encryptionkey@0"
I20260620 21:45:52.888487 8324 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:52.890425 8324 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 21:45:52.890527 8324 log.cc:1196] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 21:45:52.890792 8324 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: Deleting consensus metadata
I20260620 21:45:52.890974 8324 master_runner.cc:315] Copying system tablet from 127.1.155.190:36873
I20260620 21:45:52.892047 8324 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: tablet copy: Beginning tablet copy session from remote peer at address 127.1.155.190:36873
I20260620 21:45:52.900225 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.892649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46296 (local address 127.1.155.190:36873)
0620 21:45:52.892781 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:52.892786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:52.892994 (+ 208us) server_negotiation.cc:407] Connection header received
0620 21:45:52.893155 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:52.893158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:52.893212 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:52.893301 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:52.893713 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.894506 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.895442 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.895704 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.895812 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.897113 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:52.897116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:52.897118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:52.897156 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:52.898664 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.899086 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.899091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.899093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.899175 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:52.899435 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:52.899439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:52.899441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:52.899616 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:52.899719 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:52.899960 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 21:45:52.900065 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:45:52.900236 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:52.892587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.0.0.1:46296)
0620 21:45:52.892902 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:52.892913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:52.893041 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:52.893309 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:52.893312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:52.893511 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:52.893599 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.893604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.894609 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.894615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.895314 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:52.895323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:52.895823 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:52.895826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:52.895915 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:52.896447 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:52.896459 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:52.896979 (+ 520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:52.898784 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.898789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.898791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.898991 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.899262 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:52.899267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:52.899269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:52.899351 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:52.899707 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:52.899710 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:52.899809 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 21:45:52.899911 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 21:45:52.900056 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":80,"threads_started":1}
I20260620 21:45:52.900991 7778 tablet_copy_service.cc:140] P 32c4b8ae8381486abded8f30d55598ee: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0ec4a9136bbd4d87ac89b1d9a9b104db ({username='oryx', principal='oryx/127.1.155.188@KRBTEST.COM'} at 127.0.0.1:46296)
I20260620 21:45:52.901098 7778 tablet_copy_service.cc:161] P 32c4b8ae8381486abded8f30d55598ee: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0ec4a9136bbd4d87ac89b1d9a9b104db at {username='oryx', principal='oryx/127.1.155.188@KRBTEST.COM'} at 127.0.0.1:46296: session id = 0ec4a9136bbd4d87ac89b1d9a9b104db-00000000000000000000000000000000
I20260620 21:45:52.901971 7778 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 21:45:52.903043 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 21:45:52.909580 8324 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: tablet copy: Starting download of 0 data blocks...
I20260620 21:45:52.909785 8324 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: tablet copy: Starting download of 1 WAL segments...
I20260620 21:45:52.911167 8324 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 21:45:52.913054 7778 tablet_copy_service.cc:342] P 32c4b8ae8381486abded8f30d55598ee: Request end of tablet copy session 0ec4a9136bbd4d87ac89b1d9a9b104db-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.155.188@KRBTEST.COM'} at 127.0.0.1:46296
I20260620 21:45:52.913148 7778 tablet_copy_service.cc:433] P 32c4b8ae8381486abded8f30d55598ee: ending tablet copy session 0ec4a9136bbd4d87ac89b1d9a9b104db-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0ec4a9136bbd4d87ac89b1d9a9b104db
I20260620 21:45:52.919864 8324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:52.921494 8523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:52.921501 8526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:52.922045 8324 server_base.cc:1034] running on GCE node
W20260620 21:45:52.922024 8524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:52.922303 8324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:52.922605 8324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:52.923749 8324 hybrid_clock.cc:648] HybridClock initialized: now 1781991952923749 us; error 50 us; skew 500 ppm
Jun 20 21:45:52 dist-test-slave-5bvn krb5kdc[7713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:52.925246 8324 init.cc:377] Logged in from keytab as oryx/127.1.155.188@KRBTEST.COM (short username oryx)
I20260620 21:45:52.925695 8324 webserver.cc:460] Webserver started at http://127.1.155.188:34421/ using document root <none> and password file <none>
I20260620 21:45:52.925915 8324 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:52.925962 8324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:52.927018 8324 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:45:52.927911 8533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:52.928130 8324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:45:52.928222 8324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/wal
uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db"
format_stamp: "Formatted at 2026-06-20 21:45:52 on dist-test-slave-5bvn"
server_key: "f411a987e603051a4277fcda78a746b7"
server_key_iv: "ac9a2e171cc17d39900395d680c57027"
server_key_version: "encryptionkey@0"
I20260620 21:45:52.928366 8324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:52.939841 8324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:52.941892 8324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:52.946794 8324 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.188:40219
I20260620 21:45:52.947137 8585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.188:40219 every 8 connection(s)
I20260620 21:45:52.947448 8324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/master-2/data/info.pb
I20260620 21:45:52.948807 8586 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:45:52.949986 8586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: Bootstrap starting.
I20260620 21:45:52.955963 8586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:45:52.956244 8586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: Bootstrap complete.
I20260620 21:45:52.956713 8586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } }
I20260620 21:45:52.956851 8586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ec4a9136bbd4d87ac89b1d9a9b104db, State: Initialized, Role: LEARNER
I20260620 21:45:52.956967 8586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } }
I20260620 21:45:52.957283 8589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } } }
I20260620 21:45:52.957453 8589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: This master's current role is: LEARNER
I20260620 21:45:52.957451 8586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:52.960268 8600 catalog_manager.cc:1259] Loaded cluster ID: 4a3ea318e1ba4c3598f9fdf369f79031
I20260620 21:45:52.960317 8600 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: loading cluster ID for follower catalog manager: success
I20260620 21:45:52.962020 8600 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: acquiring CA information for follower catalog manager: success
I20260620 21:45:52.963054 8600 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:45:53.002019 8083 heartbeater.cc:503] Master 127.1.155.190:36873 was elected leader, sending a full tablet report...
W20260620 21:45:53.019075 8505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:53.019518 8505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:53.019594 8505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:53.026513 8505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:53.026600 8505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:53.026690 8505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:45:53.034644 8505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.129.keytab
--principal=oryx/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:41139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=46109
--tserver_master_addrs=127.1.155.190:36873,127.1.155.189:38729,127.1.155.188:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:53.036032 8505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:53.037267 8505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:53.048002 8608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:53.047960 8605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:53.047960 8606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:53.048275 8505 server_base.cc:1034] running on GCE node
I20260620 21:45:53.048774 8505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:53.049429 8505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:53.050614 8505 hybrid_clock.cc:648] HybridClock initialized: now 1781991953050603 us; error 38 us; skew 500 ppm
Jun 20 21:45:53 dist-test-slave-5bvn krb5kdc[7713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991953, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:53.053557 8505 init.cc:377] Logged in from keytab as oryx/127.1.155.129@KRBTEST.COM (short username oryx)
I20260620 21:45:53.054476 8505 webserver.cc:460] Webserver started at http://127.1.155.129:46109/ using document root <none> and password file <none>
I20260620 21:45:53.055042 8505 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:53.055152 8505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:53.059288 8505 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 21:45:53.061404 8615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:53.062914 8505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:45:53.063047 8505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "b2a89e130a404ae68c475898d6f89967"
format_stamp: "Formatted at 2026-06-20 21:45:51 on dist-test-slave-5bvn"
server_key: "1d40f76de17ac17504b8e9cfa85a6863"
server_key_iv: "8cf7b3af7ad8788ddff4686ad84176b3"
server_key_version: "encryptionkey@0"
I20260620 21:45:53.063647 8505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:53.075642 8505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:53.079062 8505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:53.079281 8505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:53.079902 8505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:53.080785 8505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:53.080857 8505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:53.080942 8505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:53.080994 8505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:53.090322 8505 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:41139
I20260620 21:45:53.090358 8728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:41139 every 8 connection(s)
I20260620 21:45:53.091342 8505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:45:53.092258 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 8505
I20260620 21:45:53.092478 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 7954
I20260620 21:45:53.100077 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:36809
--local_ip_for_outbound_sockets=127.1.155.130
--tserver_master_addrs=127.1.155.190:36873
--webserver_port=39183
--webserver_interface=127.1.155.130
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.130.keytab
--principal=oryx/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.155.190:36873,127.1.155.189:38729
--tserver_master_addrs=127.1.155.190:36873,127.1.155.189:38729,127.1.155.188:40219 with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:45:53 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991953, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
Jun 20 21:45:53 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991953, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.188@KRBTEST.COM
Jun 20 21:45:53 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991953, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.189@KRBTEST.COM
I20260620 21:45:53.110528 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.093444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:33575 (local address 127.1.155.190:36873)
0620 21:45:53.093599 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.093606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.095226 (+ 1620us) server_negotiation.cc:407] Connection header received
0620 21:45:53.097247 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.097250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.097309 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.097368 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.098684 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.099508 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.101149 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.101492 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.101588 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.106159 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.106163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.106166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.106201 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.107980 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.108576 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.108580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.108581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.108674 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.109107 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.109111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.109112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.109378 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.109466 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.110210 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.110361 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:45:53.111011 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.093794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:43145 (local address 127.1.155.189:38729)
0620 21:45:53.093932 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.093937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.095443 (+ 1506us) server_negotiation.cc:407] Connection header received
0620 21:45:53.096597 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.096604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.096672 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.096775 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.098686 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.099515 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.102049 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.102414 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.102479 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.106936 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.106942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.106947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.106985 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.108849 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.109426 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.109430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.109434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.109528 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.109899 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.109903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.109906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.110157 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.110265 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.110656 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.110790 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 21:45:53.112018 8736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.095398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:33539 (local address 127.1.155.188:40219)
0620 21:45:53.096045 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.096049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.096064 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:45:53.096686 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.096689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.096769 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.096858 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.098693 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.099812 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.101373 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.101775 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.101909 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.106139 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.106145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.106151 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.106190 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.109484 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.109958 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.109961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.109963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.110046 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.110328 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.110331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.110334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.110525 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.110614 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.111746 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.111877 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":101,"threads_started":1}
I20260620 21:45:53.112452 8733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.093789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.129:33575)
0620 21:45:53.095061 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:53.095114 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:53.096337 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:53.097621 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:53.097624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:53.097781 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:53.098507 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.098526 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.099697 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.099700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.101014 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.101023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.101755 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.101759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.102048 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.102722 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:53.102743 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:53.105998 (+ 3255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:53.108115 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.108122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.108159 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.108460 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.108775 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.108781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.108786 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.108985 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.109536 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:53.109546 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:53.109914 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 21:45:53.110974 (+ 1060us) client_negotiation.cc:240] Negotiation successful
0620 21:45:53.111496 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":577,"spinlock_wait_cycles":9344,"thread_start_us":132,"threads_started":1}
I20260620 21:45:53.112506 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.093789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:40219 (local address 127.1.155.129:33539)
0620 21:45:53.095803 (+ 2014us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:53.095816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:53.096536 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:53.097010 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:53.097020 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:53.097722 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:53.098509 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.098529 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.099951 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.099954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.101203 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.101217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.102052 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.102057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.102190 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.102882 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:53.102901 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:53.105902 (+ 3001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:53.109590 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.109593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.109595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.109822 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.110149 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.110152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.110154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.110230 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.110655 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:53.110658 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:53.110775 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 21:45:53.110885 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 21:45:53.111278 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1933,"mutex_wait_us":1080,"spinlock_wait_cycles":10624,"thread_start_us":89,"threads_started":1}
I20260620 21:45:53.112865 8734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.094342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38729 (local address 127.1.155.129:43145)
0620 21:45:53.095355 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:53.095380 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:53.096337 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:53.097010 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:53.097020 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:53.097632 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:53.098507 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.098526 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.099697 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.099700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.101917 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.101925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.102677 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.102681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.102776 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.103174 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:53.103188 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:53.106754 (+ 3566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:53.108994 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.109000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.109005 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.109297 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.109654 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.109658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.109661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.109755 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.110332 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:53.110335 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:53.110446 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 21:45:53.111159 (+ 713us) client_negotiation.cc:240] Negotiation successful
0620 21:45:53.111425 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":1204,"spinlock_wait_cycles":32640,"thread_start_us":92,"threads_started":1}
I20260620 21:45:53.114079 8731 heartbeater.cc:347] Connected to a master server at 127.1.155.189:38729
I20260620 21:45:53.114115 8729 heartbeater.cc:347] Connected to a master server at 127.1.155.188:40219
I20260620 21:45:53.114117 8730 heartbeater.cc:347] Connected to a master server at 127.1.155.190:36873
I20260620 21:45:53.114320 8730 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:53.114319 8729 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:53.114320 8731 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:53.114928 8729 heartbeater.cc:511] Master 127.1.155.188:40219 requested a full tablet report, sending...
I20260620 21:45:53.114943 8731 heartbeater.cc:511] Master 127.1.155.189:38729 requested a full tablet report, sending...
I20260620 21:45:53.114929 8730 heartbeater.cc:511] Master 127.1.155.190:36873 requested a full tablet report, sending...
I20260620 21:45:53.116189 7758 ts_manager.cc:194] Re-registered known tserver with Master: b2a89e130a404ae68c475898d6f89967 (127.1.155.129:41139)
I20260620 21:45:53.116420 8550 ts_manager.cc:194] Registered new tserver with Master: b2a89e130a404ae68c475898d6f89967 (127.1.155.129:41139)
I20260620 21:45:53.116923 7758 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:33575
I20260620 21:45:53.117731 8360 ts_manager.cc:194] Registered new tserver with Master: b2a89e130a404ae68c475898d6f89967 (127.1.155.129:41139)
W20260620 21:45:53.216641 8737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:53.216913 8737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:53.216956 8737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:53.221086 8737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:53.221146 8737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:53.221244 8737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:45:53.226450 8737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.130.keytab
--principal=oryx/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:36809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=39183
--tserver_master_addrs=127.1.155.190:36873,127.1.155.189:38729,127.1.155.188:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:53.227511 8737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:53.228474 8737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:53.238076 8745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:53.238179 8737 server_base.cc:1034] running on GCE node
W20260620 21:45:53.238067 8742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:53.238067 8743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:53.238808 8737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:53.239496 8737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:53.240365 8217 heartbeater.cc:503] Master 127.1.155.190:36873 was elected leader, sending a full tablet report...
I20260620 21:45:53.240687 8737 hybrid_clock.cc:648] HybridClock initialized: now 1781991953240654 us; error 48 us; skew 500 ppm
Jun 20 21:45:53 dist-test-slave-5bvn krb5kdc[7713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991953, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:53.243829 8737 init.cc:377] Logged in from keytab as oryx/127.1.155.130@KRBTEST.COM (short username oryx)
I20260620 21:45:53.244704 8737 webserver.cc:460] Webserver started at http://127.1.155.130:39183/ using document root <none> and password file <none>
I20260620 21:45:53.245244 8737 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:53.245332 8737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:53.249155 8737 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:45:53.251330 8752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:53.252563 8737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:45:53.252686 8737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "ab51c4cca72e4d74a82e11db2a47fc40"
format_stamp: "Formatted at 2026-06-20 21:45:51 on dist-test-slave-5bvn"
server_key: "8bce0e4eb96fd31402233d01fa19741e"
server_key_iv: "a9f09b7e0bd71fc348fcf64b708fad45"
server_key_version: "encryptionkey@0"
I20260620 21:45:53.253202 8737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:53.274814 8737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:53.277998 8737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:53.278183 8737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:53.278772 8737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:53.279616 8737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:53.279661 8737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:53.279727 8737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:53.279758 8737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:53.288427 8737 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:36809
I20260620 21:45:53.288503 8865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:36809 every 8 connection(s)
I20260620 21:45:53.289453 8737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:45:53.298472 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 8737
I20260620 21:45:53.298659 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 8088
Jun 20 21:45:53 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991953, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.188@KRBTEST.COM
Jun 20 21:45:53 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991953, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.189@KRBTEST.COM
Jun 20 21:45:53 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991953, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
I20260620 21:45:53.305585 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.291228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:56337 (local address 127.1.155.189:38729)
0620 21:45:53.291399 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.291406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.293405 (+ 1999us) server_negotiation.cc:407] Connection header received
0620 21:45:53.293588 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.293594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.293647 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.293698 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.295006 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.295818 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.297122 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.297353 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.297438 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.300976 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.300982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.300989 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.301022 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.303048 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.303677 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.303681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.303683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.303781 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.304190 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.304194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.304196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.304443 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.304514 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.305284 (+ 770us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.305414 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:45:53.305585 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.291216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:44949 (local address 127.1.155.190:36873)
0620 21:45:53.291346 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.291351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.292320 (+ 969us) server_negotiation.cc:407] Connection header received
0620 21:45:53.293319 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.293322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.293387 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.293481 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.295006 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.295801 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.297111 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.297358 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.297431 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.301609 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.301613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.301615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.301649 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.303562 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.304058 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.304062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.304064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.304150 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.304449 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.304452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.304454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.304661 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.304726 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.305317 (+ 591us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.305424 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:45:53.305794 8736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.291544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:46317 (local address 127.1.155.188:40219)
0620 21:45:53.291684 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.291688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.292316 (+ 628us) server_negotiation.cc:407] Connection header received
0620 21:45:53.293311 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.293314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.293374 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.293491 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.294987 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.295629 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.296780 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.297015 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.297130 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.301224 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.301228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.301231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.301259 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.303542 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.304209 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.304215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.304220 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.304326 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.304655 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.304662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.304667 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.304907 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.305001 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.305455 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.305602 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:45:53.307431 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.291833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:40219 (local address 127.1.155.130:46317)
0620 21:45:53.292166 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:53.292220 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:53.293112 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:53.293668 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:53.293675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:53.294224 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:53.294850 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.294863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.295770 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.295772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.296624 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.296632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.297556 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.297558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.297694 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.298352 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:53.298378 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:53.301082 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:53.303670 (+ 2588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.303676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.303681 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.304098 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.304450 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.304456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.304461 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.304554 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.305106 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:53.305150 (+ 44us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:53.305270 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 21:45:53.306568 (+ 1298us) client_negotiation.cc:240] Negotiation successful
0620 21:45:53.306742 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":7,"spinlock_wait_cycles":4480,"thread_start_us":149,"threads_started":1}
I20260620 21:45:53.307428 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.291711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.130:44949)
0620 21:45:53.292167 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:53.292220 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:53.293086 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:53.293670 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:53.293678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:53.294251 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:53.294860 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.294868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.295944 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.295946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.296994 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.297003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.297603 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.297606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.297806 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.298589 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:53.298603 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:53.301427 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:53.303672 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.303675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.303676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.303955 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.304274 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.304277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.304279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.304349 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.304793 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:53.304796 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:53.304963 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 21:45:53.305749 (+ 786us) client_negotiation.cc:240] Negotiation successful
0620 21:45:53.306226 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":312,"spinlock_wait_cycles":52992,"thread_start_us":124,"threads_started":1}
I20260620 21:45:53.307684 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.292599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38729 (local address 127.1.155.130:56337)
0620 21:45:53.293320 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:53.293333 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:53.293477 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:53.293862 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:53.293867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:53.294268 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:53.294860 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.294868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.295975 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.295977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.296994 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.297003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.297598 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.297602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.297767 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.298677 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:53.298697 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:53.300736 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:53.303183 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.303189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.303206 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.303557 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.303903 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.303907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.303910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.304085 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.304594 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:53.304600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:53.304963 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 21:45:53.305788 (+ 825us) client_negotiation.cc:240] Negotiation successful
0620 21:45:53.306268 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":507,"spinlock_wait_cycles":7808,"thread_start_us":530,"threads_started":1}
I20260620 21:45:53.308786 8867 heartbeater.cc:347] Connected to a master server at 127.1.155.190:36873
I20260620 21:45:53.308801 8866 heartbeater.cc:347] Connected to a master server at 127.1.155.188:40219
I20260620 21:45:53.309012 8866 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:53.309038 8868 heartbeater.cc:347] Connected to a master server at 127.1.155.189:38729
I20260620 21:45:53.309092 8868 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:53.309242 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:41081
--local_ip_for_outbound_sockets=127.1.155.131
--tserver_master_addrs=127.1.155.190:36873
--webserver_port=45425
--webserver_interface=127.1.155.131
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.131.keytab
--principal=oryx/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.155.190:36873,127.1.155.189:38729
--tserver_master_addrs=127.1.155.190:36873,127.1.155.189:38729,127.1.155.188:40219 with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:45:53.309489 8866 heartbeater.cc:511] Master 127.1.155.188:40219 requested a full tablet report, sending...
I20260620 21:45:53.309013 8867 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:53.309989 8867 heartbeater.cc:511] Master 127.1.155.190:36873 requested a full tablet report, sending...
I20260620 21:45:53.309491 8868 heartbeater.cc:511] Master 127.1.155.189:38729 requested a full tablet report, sending...
I20260620 21:45:53.310667 8550 ts_manager.cc:194] Registered new tserver with Master: ab51c4cca72e4d74a82e11db2a47fc40 (127.1.155.130:36809)
I20260620 21:45:53.310911 8360 ts_manager.cc:194] Registered new tserver with Master: ab51c4cca72e4d74a82e11db2a47fc40 (127.1.155.130:36809)
I20260620 21:45:53.311185 7758 ts_manager.cc:194] Re-registered known tserver with Master: ab51c4cca72e4d74a82e11db2a47fc40 (127.1.155.130:36809)
I20260620 21:45:53.311827 7758 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:44949
I20260620 21:45:53.409770 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.405792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:40219 (local address 127.0.0.1:46298)
0620 21:45:53.406125 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:53.406138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:53.406247 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:53.406529 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:53.406531 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:53.406544 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:53.406638 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.406642 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.407696 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.407699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.408483 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.408492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.409415 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.409418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.409493 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.409515 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:45:53.409572 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 21:45:53.409646 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":103,"threads_started":1}
I20260620 21:45:53.409806 8736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.405942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.1.155.188:40219)
0620 21:45:53.406097 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.406103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.406223 (+ 120us) server_negotiation.cc:407] Connection header received
0620 21:45:53.406333 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.406336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.406398 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.406515 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:53.406766 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.407557 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.408621 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.409277 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.409397 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.409549 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.409597 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.409667 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:45:53.410509 8560 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 21:45:53.410781 8589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32c4b8ae8381486abded8f30d55598ee. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } } }
I20260620 21:45:53.410944 8589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: This master's current role is: LEARNER
W20260620 21:45:53.422915 8873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:53.423156 8873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:53.423228 8873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:53.427250 8873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:53.427309 8873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:53.427397 8873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:45:53.432695 8873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.131.keytab
--principal=oryx/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:41081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=45425
--tserver_master_addrs=127.1.155.190:36873,127.1.155.189:38729,127.1.155.188:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:53.433673 8873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:53.434828 8873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:53.444672 8882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:53.444682 8880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:53.444682 8879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:53.444952 8873 server_base.cc:1034] running on GCE node
I20260620 21:45:53.445407 8873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:53.446089 8873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:53.447292 8873 hybrid_clock.cc:648] HybridClock initialized: now 1781991953447273 us; error 40 us; skew 500 ppm
Jun 20 21:45:53 dist-test-slave-5bvn krb5kdc[7713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991953, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:53.450027 8873 init.cc:377] Logged in from keytab as oryx/127.1.155.131@KRBTEST.COM (short username oryx)
I20260620 21:45:53.450858 8873 webserver.cc:460] Webserver started at http://127.1.155.131:45425/ using document root <none> and password file <none>
I20260620 21:45:53.451391 8873 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:53.451479 8873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:53.455181 8873 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:45:53.457132 8889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:53.458552 8873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:45:53.458706 8873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "231e4941b8d34841bd74c6b0aea79ab4"
format_stamp: "Formatted at 2026-06-20 21:45:52 on dist-test-slave-5bvn"
server_key: "92501d7d0aeaf61543869d5593af8e28"
server_key_iv: "cb8ccfe897c44e167f5eb208ae286849"
server_key_version: "encryptionkey@0"
I20260620 21:45:53.459271 8873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:53.478226 8873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:53.481256 8873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:53.481458 8873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:53.482071 8873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:45:53.482913 8873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:53.482983 8873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:53.483054 8873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:53.483098 8873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:53.491791 8873 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:41081
I20260620 21:45:53.491814 9002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:41081 every 8 connection(s)
I20260620 21:45:53.492753 8873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:45:53.496563 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 8873
Jun 20 21:45:53 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991953, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.188@KRBTEST.COM
Jun 20 21:45:53 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991953, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
Jun 20 21:45:53 dist-test-slave-5bvn krb5kdc[7713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991953, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.189@KRBTEST.COM
I20260620 21:45:53.509917 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.499762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46308 (local address 127.1.155.190:36873)
0620 21:45:53.499994 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.500001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.500326 (+ 325us) server_negotiation.cc:407] Connection header received
0620 21:45:53.500482 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.500485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.500528 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.500586 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.501457 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.502230 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.503492 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.503829 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.503902 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.505946 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.505953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.505958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.505987 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.507868 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.508390 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.508396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.508401 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.508505 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.508837 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.508842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.508843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.509065 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.509168 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.509560 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.509688 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":82,"threads_started":1}
I20260620 21:45:53.510250 8736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.494985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:40733 (local address 127.1.155.188:40219)
0620 21:45:53.495099 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.495103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.495769 (+ 666us) server_negotiation.cc:407] Connection header received
0620 21:45:53.496907 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.496910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.496986 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.497083 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.499004 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.499682 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.501609 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.501980 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.502053 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.505690 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.505694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.505696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.505743 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.507868 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.508466 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.508469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.508470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.508547 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.508953 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.508957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.508958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.509148 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.509213 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.509990 (+ 777us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.510110 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 21:45:53.512264 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.495229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:38331 (local address 127.1.155.189:38729)
0620 21:45:53.495369 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.495374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.496121 (+ 747us) server_negotiation.cc:407] Connection header received
0620 21:45:53.497009 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.497016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.497074 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.497142 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.500459 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.501279 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.502873 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.503222 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.503328 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.508238 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.508258 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.508266 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.508300 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.509957 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.510446 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.510451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.510453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.510538 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.511149 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.511153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.511154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.511364 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.511460 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.511973 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.512091 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:45:53.512301 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.495027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38729 (local address 127.1.155.131:38331)
0620 21:45:53.496017 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:53.496029 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:53.496675 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:53.497394 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:53.497403 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:53.498111 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:53.500321 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.500333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.501404 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.501407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.502726 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.502738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.504176 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.504183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.504305 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.505862 (+ 1557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:53.505878 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:53.508074 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:53.510054 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.510059 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.510064 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.510329 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.510641 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.510646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.510650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.510734 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.511503 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:53.511506 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:53.511588 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 21:45:53.511673 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 21:45:53.511797 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":1673,"spinlock_wait_cycles":22656,"thread_start_us":135,"threads_started":1}
I20260620 21:45:53.511864 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.495027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:40219 (local address 127.1.155.131:40733)
0620 21:45:53.495618 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:53.495669 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:53.496675 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:53.497396 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:53.497405 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:53.498227 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:53.498853 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.498879 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.500037 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.500041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.501489 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.501498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.502226 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.502229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.502459 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.503513 (+ 1054us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:53.503532 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:53.505504 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:53.507999 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.508005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.508022 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.508359 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.508657 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.508663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.508667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.508851 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.509293 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:53.509299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:53.509679 (+ 380us) client_negotiation.cc:769] Sending connection context
0620 21:45:53.510458 (+ 779us) client_negotiation.cc:240] Negotiation successful
0620 21:45:53.510787 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":76,"spinlock_wait_cycles":19072,"thread_start_us":148,"threads_started":1}
I20260620 21:45:53.513854 9003 heartbeater.cc:347] Connected to a master server at 127.1.155.188:40219
I20260620 21:45:53.513991 9005 heartbeater.cc:347] Connected to a master server at 127.1.155.189:38729
I20260620 21:45:53.514086 9005 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:53.514084 9003 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:53.514508 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.494655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:49601 (local address 127.1.155.190:36873)
0620 21:45:53.494791 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.494796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.496343 (+ 1547us) server_negotiation.cc:407] Connection header received
0620 21:45:53.497073 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.497077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.497131 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.497181 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.499023 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.499786 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.503231 (+ 3445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.503572 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.503660 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.507505 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.507509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.507512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.507540 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.510719 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.513046 (+ 2327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.513049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.513051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.513133 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.513450 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.513453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.513455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.513639 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.513726 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.514264 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.514370 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:45:53.514707 9003 heartbeater.cc:511] Master 127.1.155.188:40219 requested a full tablet report, sending...
I20260620 21:45:53.514834 9007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.495027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.131:49601)
0620 21:45:53.496245 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:53.496257 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:53.496956 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:53.497394 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:53.497403 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:53.498130 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:53.498854 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.498874 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.499947 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.499951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.503112 (+ 3161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.503121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.503805 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.503809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.503900 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.504519 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:53.504538 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:53.507348 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:53.510820 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.510823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.510825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.511033 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.513240 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:53.513243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:53.513256 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:53.513348 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:53.513777 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:53.513781 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:53.513912 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 21:45:53.514009 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 21:45:53.514143 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1143,"mutex_wait_us":2550,"thread_start_us":85,"threads_started":1}
I20260620 21:45:53.515358 9004 heartbeater.cc:347] Connected to a master server at 127.1.155.190:36873
I20260620 21:45:53.515432 9004 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:53.515568 9004 heartbeater.cc:511] Master 127.1.155.190:36873 requested a full tablet report, sending...
I20260620 21:45:53.515892 8550 ts_manager.cc:194] Registered new tserver with Master: 231e4941b8d34841bd74c6b0aea79ab4 (127.1.155.131:41081)
I20260620 21:45:53.516170 7752 ts_manager.cc:194] Re-registered known tserver with Master: 231e4941b8d34841bd74c6b0aea79ab4 (127.1.155.131:41081)
I20260620 21:45:53.516872 7752 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:49601
I20260620 21:45:53.513664 7758 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 21:45:53.514705 9005 heartbeater.cc:511] Master 127.1.155.189:38729 requested a full tablet report, sending...
I20260620 21:45:53.518591 8360 ts_manager.cc:194] Registered new tserver with Master: 231e4941b8d34841bd74c6b0aea79ab4 (127.1.155.131:41081)
W20260620 21:45:53.518694 7758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:53.521878 8319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: NON_VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: true } } }
I20260620 21:45:53.522012 8319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:53.522322 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.498737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.1.155.188:40219)
0620 21:45:53.500302 (+ 1565us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.500306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.500318 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:45:53.500353 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.500358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.500415 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.501365 (+ 950us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.501457 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.502260 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.503777 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.504805 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.507070 (+ 2265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.507193 (+ 123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.507197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.507199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.507218 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.520002 (+ 12784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.520589 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.520593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.520594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.520705 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.521032 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.521034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.521036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.521260 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.521932 (+ 672us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.522016 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.522151 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1489,"thread_start_us":69,"threads_started":1}
I20260620 21:45:53.523277 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.499860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.1.155.189:38729)
0620 21:45:53.508677 (+ 8817us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.508683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.508702 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:45:53.508740 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.508742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.508792 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.509668 (+ 876us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.509780 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.511354 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.512454 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.512814 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.512916 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.517151 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.517155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.517157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.517180 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.520121 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.520724 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.520730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.520735 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.520838 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.521157 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.521160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.521162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.521376 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.522920 (+ 1544us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.523002 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.523120 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8728,"thread_start_us":85,"threads_started":1}
I20260620 21:45:53.524901 8319 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee: attempting to promote NON_VOTER 0ec4a9136bbd4d87ac89b1d9a9b104db to VOTER
I20260620 21:45:53.525178 8319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } }
I20260620 21:45:53.526871 8560 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 LEARNER]: Refusing update from remote peer 32c4b8ae8381486abded8f30d55598ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 21:45:53.527379 9021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 21:45:53.528189 8370 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 FOLLOWER]: Refusing update from remote peer 32c4b8ae8381486abded8f30d55598ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 21:45:53.528549 9021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 21:45:53.529088 9021 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0ec4a9136bbd4d87ac89b1d9a9b104db (127.1.155.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } } }
I20260620 21:45:53.529650 8560 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0ec4a9136bbd4d87ac89b1d9a9b104db (127.1.155.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } } }
I20260620 21:45:53.530638 9022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } } }
I20260620 21:45:53.530825 9022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:53.530659 9021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } } }
I20260620 21:45:53.531448 9021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:53.531539 8589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } } }
I20260620 21:45:53.531648 8589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:45:53.532070 9022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } } }
I20260620 21:45:53.532172 9022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:53.532372 8370 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0ec4a9136bbd4d87ac89b1d9a9b104db (127.1.155.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } } }
I20260620 21:45:53.535692 9030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32c4b8ae8381486abded8f30d55598ee" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } } }
I20260620 21:45:53.535840 9030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:45:53.539798 9032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.533927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:41139 (local address 127.0.0.1:42088)
0620 21:45:53.534769 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:53.534781 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:53.534870 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:53.535455 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:53.535458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:53.535478 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:53.535764 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.535773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.537248 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.537253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.538363 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.538372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.539422 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.539425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.539500 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.539522 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:45:53.539580 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 21:45:53.539652 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":22,"thread_start_us":91,"threads_started":1}
I20260620 21:45:53.540434 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.534352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42088 (local address 127.1.155.129:41139)
0620 21:45:53.534652 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.534664 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.534968 (+ 304us) server_negotiation.cc:407] Connection header received
0620 21:45:53.535058 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.535063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.535307 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.535470 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:53.535953 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.537137 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.538498 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.539331 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.539426 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.539578 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.539629 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.540296 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":126,"threads_started":1}
I20260620 21:45:53.540488 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.533622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:36809 (local address 127.0.0.1:58206)
0620 21:45:53.534492 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:53.534507 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:53.534595 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:53.536576 (+ 1981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:53.536579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:53.536596 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:53.536683 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.536688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.538302 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.538307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.539048 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.539056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.540094 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.540099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.540184 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.540220 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 21:45:53.540284 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 21:45:53.540356 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260620 21:45:53.540620 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.533321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:41081 (local address 127.0.0.1:40436)
0620 21:45:53.534034 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:53.534049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:53.534121 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:53.536342 (+ 2221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:53.536345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:53.536363 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:53.536467 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.536473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.538111 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.538114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.539249 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.539260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.540221 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.540224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.540336 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.540363 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:45:53.540412 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 21:45:53.540491 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":1}
I20260620 21:45:53.541126 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.534764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.1.155.130:36809)
0620 21:45:53.536085 (+ 1321us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.536095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.536116 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:45:53.536183 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.536188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.536350 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.536480 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:53.537072 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.538193 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.539173 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.539944 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.540511 (+ 567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.540795 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.540871 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.540960 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1232,"thread_start_us":88,"threads_started":1}
I20260620 21:45:53.541867 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.534423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.1.155.131:41081)
0620 21:45:53.535807 (+ 1384us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.535822 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.535848 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:45:53.535965 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.535971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.536162 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.536632 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:53.536808 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.537999 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.539376 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.540102 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.540898 (+ 796us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.541020 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.541063 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.541153 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":96,"threads_started":1}
I20260620 21:45:53.543540 8663 tablet_service.cc:1467] Processing CreateTablet for tablet d0ff5195dd8e4c0187a3704f0720d59a (DEFAULT_TABLE table=test-table [id=6e6382f28e9f4aea825c4e17b311f4e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:53.543542 8800 tablet_service.cc:1467] Processing CreateTablet for tablet d0ff5195dd8e4c0187a3704f0720d59a (DEFAULT_TABLE table=test-table [id=6e6382f28e9f4aea825c4e17b311f4e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:53.544206 8937 tablet_service.cc:1467] Processing CreateTablet for tablet d0ff5195dd8e4c0187a3704f0720d59a (DEFAULT_TABLE table=test-table [id=6e6382f28e9f4aea825c4e17b311f4e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:53.544628 8663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ff5195dd8e4c0187a3704f0720d59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:53.544721 8800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ff5195dd8e4c0187a3704f0720d59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:53.545083 8937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ff5195dd8e4c0187a3704f0720d59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:53.551502 9036 tablet_bootstrap.cc:492] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40: Bootstrap starting.
I20260620 21:45:53.552598 9037 tablet_bootstrap.cc:492] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4: Bootstrap starting.
I20260620 21:45:53.554162 9036 tablet_bootstrap.cc:654] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:53.554154 9038 tablet_bootstrap.cc:492] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967: Bootstrap starting.
I20260620 21:45:53.554703 9037 tablet_bootstrap.cc:654] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:53.555059 9036 log.cc:824] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:53.555486 9037 log.cc:824] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:53.556010 9038 tablet_bootstrap.cc:654] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:53.556761 9038 log.cc:824] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:53.557327 9036 tablet_bootstrap.cc:492] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40: No bootstrap required, opened a new log
I20260620 21:45:53.557523 9036 ts_tablet_manager.cc:1397] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:45:53.558610 9038 tablet_bootstrap.cc:492] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967: No bootstrap required, opened a new log
I20260620 21:45:53.558755 9038 ts_tablet_manager.cc:1397] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:45:53.561214 9036 raft_consensus.cc:348] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } }
I20260620 21:45:53.561465 9036 raft_consensus.cc:374] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:53.561550 9036 raft_consensus.cc:729] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab51c4cca72e4d74a82e11db2a47fc40, State: Initialized, Role: FOLLOWER
I20260620 21:45:53.562027 9036 consensus_queue.cc:260] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } }
I20260620 21:45:53.562019 9038 raft_consensus.cc:348] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } }
I20260620 21:45:53.562238 9038 raft_consensus.cc:374] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:53.562294 9038 raft_consensus.cc:729] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a89e130a404ae68c475898d6f89967, State: Initialized, Role: FOLLOWER
I20260620 21:45:53.562759 9038 consensus_queue.cc:260] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } }
I20260620 21:45:53.563320 9036 ts_tablet_manager.cc:1428] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:45:53.563596 9038 ts_tablet_manager.cc:1428] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 21:45:53.563910 8730 heartbeater.cc:503] Master 127.1.155.190:36873 was elected leader, sending a full tablet report...
I20260620 21:45:53.564164 8867 heartbeater.cc:503] Master 127.1.155.190:36873 was elected leader, sending a full tablet report...
I20260620 21:45:53.570549 9037 tablet_bootstrap.cc:492] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4: No bootstrap required, opened a new log
I20260620 21:45:53.570791 9037 ts_tablet_manager.cc:1397] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20260620 21:45:53.574591 9037 raft_consensus.cc:348] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } }
I20260620 21:45:53.574853 9037 raft_consensus.cc:374] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:53.574942 9037 raft_consensus.cc:729] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231e4941b8d34841bd74c6b0aea79ab4, State: Initialized, Role: FOLLOWER
I20260620 21:45:53.575392 9037 consensus_queue.cc:260] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } }
I20260620 21:45:53.576079 9004 heartbeater.cc:503] Master 127.1.155.190:36873 was elected leader, sending a full tablet report...
I20260620 21:45:53.576344 9037 ts_tablet_manager.cc:1428] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
W20260620 21:45:53.595667 8732 tablet.cc:2369] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:45:53.714000 9043 raft_consensus.cc:482] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:53.714234 9043 raft_consensus.cc:504] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } }
I20260620 21:45:53.715278 9043 leader_election.cc:288] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231e4941b8d34841bd74c6b0aea79ab4 (127.1.155.131:41081), ab51c4cca72e4d74a82e11db2a47fc40 (127.1.155.130:36809)
I20260620 21:45:53.719295 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.715742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:58099 (local address 127.1.155.131:41081)
0620 21:45:53.715896 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.715901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.716047 (+ 146us) server_negotiation.cc:407] Connection header received
0620 21:45:53.716218 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.716221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.716271 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.716323 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:53.716767 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.717290 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.718359 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.718750 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.718879 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.718930 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.719104 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.719182 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:45:53.719317 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.715633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:41081 (local address 127.1.155.129:58099)
0620 21:45:53.715926 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:53.715941 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:53.716076 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:53.716476 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:53.716479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:53.716498 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:53.716640 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.716648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.717407 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.717412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.718232 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.718240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.718872 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.718875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.718973 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.719001 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:45:53.719088 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 21:45:53.719161 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":3,"thread_start_us":100,"threads_started":1}
I20260620 21:45:53.719789 9046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.715997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:36809 (local address 127.1.155.129:55725)
0620 21:45:53.716253 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:53.716272 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:53.716350 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:53.716572 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:53.716574 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:53.716589 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:53.716713 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.716718 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.717528 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.717535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.718446 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:53.718455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.719155 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.719159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:53.719276 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.719294 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 21:45:53.719575 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 21:45:53.719651 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":19,"thread_start_us":76,"threads_started":1}
I20260620 21:45:53.719787 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.716178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:55725 (local address 127.1.155.130:36809)
0620 21:45:53.716309 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.716315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.716329 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:45:53.716432 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.716435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.716479 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.716552 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:53.716850 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.717408 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.718593 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.719022 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.719166 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.719216 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.719604 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.719677 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:45:53.719975 8957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ff5195dd8e4c0187a3704f0720d59a" candidate_uuid: "b2a89e130a404ae68c475898d6f89967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231e4941b8d34841bd74c6b0aea79ab4" is_pre_election: true
I20260620 21:45:53.720299 8957 raft_consensus.cc:2454] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a89e130a404ae68c475898d6f89967 in term 0.
I20260620 21:45:53.720507 8820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ff5195dd8e4c0187a3704f0720d59a" candidate_uuid: "b2a89e130a404ae68c475898d6f89967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" is_pre_election: true
I20260620 21:45:53.720765 8618 leader_election.cc:302] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231e4941b8d34841bd74c6b0aea79ab4, b2a89e130a404ae68c475898d6f89967; no voters:
I20260620 21:45:53.720816 8820 raft_consensus.cc:2454] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a89e130a404ae68c475898d6f89967 in term 0.
I20260620 21:45:53.721089 9043 raft_consensus.cc:2781] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:53.721238 9043 raft_consensus.cc:482] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:53.721299 9043 raft_consensus.cc:3037] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:53.722330 9043 raft_consensus.cc:504] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } }
I20260620 21:45:53.722666 9043 leader_election.cc:288] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [CANDIDATE]: Term 1 election: Requested vote from peers 231e4941b8d34841bd74c6b0aea79ab4 (127.1.155.131:41081), ab51c4cca72e4d74a82e11db2a47fc40 (127.1.155.130:36809)
I20260620 21:45:53.723097 8820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ff5195dd8e4c0187a3704f0720d59a" candidate_uuid: "b2a89e130a404ae68c475898d6f89967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab51c4cca72e4d74a82e11db2a47fc40"
I20260620 21:45:53.723104 8957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ff5195dd8e4c0187a3704f0720d59a" candidate_uuid: "b2a89e130a404ae68c475898d6f89967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231e4941b8d34841bd74c6b0aea79ab4"
I20260620 21:45:53.723238 8820 raft_consensus.cc:3037] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:53.723233 8957 raft_consensus.cc:3037] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:53.724622 8820 raft_consensus.cc:2454] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a89e130a404ae68c475898d6f89967 in term 1.
I20260620 21:45:53.724617 8957 raft_consensus.cc:2454] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a89e130a404ae68c475898d6f89967 in term 1.
I20260620 21:45:53.724912 8618 leader_election.cc:302] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231e4941b8d34841bd74c6b0aea79ab4, b2a89e130a404ae68c475898d6f89967; no voters:
I20260620 21:45:53.725097 9043 raft_consensus.cc:2781] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:53.725289 9043 raft_consensus.cc:686] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [term 1 LEADER]: Becoming Leader. State: Replica: b2a89e130a404ae68c475898d6f89967, State: Running, Role: LEADER
I20260620 21:45:53.725669 9043 consensus_queue.cc:237] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } }
I20260620 21:45:53.729271 7757 catalog_manager.cc:5495] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2a89e130a404ae68c475898d6f89967 (127.1.155.129). New cstate: current_term: 1 leader_uuid: "b2a89e130a404ae68c475898d6f89967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } health_report { overall_health: HEALTHY } } }
W20260620 21:45:53.744423 9006 tablet.cc:2369] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:45:53.780089 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.776597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.1.155.129:41139)
0620 21:45:53.776731 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.776735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.776749 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:45:53.776819 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.776821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.776871 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.776957 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:53.777219 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.777705 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.778679 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.778891 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.778968 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.779270 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:53.779665 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:53.779759 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.779874 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.779920 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:45:53.785998 8820 raft_consensus.cc:1261] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40 [term 1 FOLLOWER]: Refusing update from remote peer b2a89e130a404ae68c475898d6f89967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:53.786093 8957 raft_consensus.cc:1261] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4 [term 1 FOLLOWER]: Refusing update from remote peer b2a89e130a404ae68c475898d6f89967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:53.786820 9043 consensus_queue.cc:1035] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:53.787150 9047 consensus_queue.cc:1035] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 21:45:53.791692 8869 tablet.cc:2369] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:45:53.798427 9051 mvcc.cc:205] Tried to move back new op lower bound from 7299039042700156928 to 7299039042463793152. Current Snapshot: MvccSnapshot[applied={T|T < 7299039042700156928}]
I20260620 21:45:53.809805 7758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:53.810038 7758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:53.814145 7758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32c4b8ae8381486abded8f30d55598ee: Sending DeleteTablet for 3 replicas of tablet d0ff5195dd8e4c0187a3704f0720d59a
I20260620 21:45:53.815088 8800 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ff5195dd8e4c0187a3704f0720d59a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:53 UTC) from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:58206
I20260620 21:45:53.815253 8937 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ff5195dd8e4c0187a3704f0720d59a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:53 UTC) from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:40436
I20260620 21:45:53.815920 8663 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ff5195dd8e4c0187a3704f0720d59a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:53 UTC) from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:42088
I20260620 21:45:53.816702 9064 tablet_replica.cc:330] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967: stopping tablet replica
I20260620 21:45:53.816862 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 7726
I20260620 21:45:53.817875 9062 tablet_replica.cc:330] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4: stopping tablet replica
I20260620 21:45:53.818249 9062 raft_consensus.cc:2229] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:53.818621 9062 raft_consensus.cc:2258] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:53.818972 9064 raft_consensus.cc:2229] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:45:53.819402 9064 raft_consensus.cc:2258] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:53.820901 9062 ts_tablet_manager.cc:1905] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:53.820900 9064 ts_tablet_manager.cc:1905] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:53.824307 9062 ts_tablet_manager.cc:1918] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:45:53.824440 9062 log.cc:1196] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/d0ff5195dd8e4c0187a3704f0720d59a
I20260620 21:45:53.824625 9064 ts_tablet_manager.cc:1918] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:45:53.825003 9062 ts_tablet_manager.cc:1939] T d0ff5195dd8e4c0187a3704f0720d59a P 231e4941b8d34841bd74c6b0aea79ab4: Deleting consensus metadata
I20260620 21:45:53.825237 9064 log.cc:1196] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-0/wal/wals/d0ff5195dd8e4c0187a3704f0720d59a
I20260620 21:45:53.825714 9064 ts_tablet_manager.cc:1939] T d0ff5195dd8e4c0187a3704f0720d59a P b2a89e130a404ae68c475898d6f89967: Deleting consensus metadata
W20260620 21:45:53.826234 8893 connection.cc:441] server connection from 127.0.0.1:40436 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40436 (request call id 1) could send its response
W20260620 21:45:53.827033 8616 connection.cc:441] server connection from 127.0.0.1:42088 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42088 (request call id 1) could send its response
I20260620 21:45:53.828398 9061 tablet_replica.cc:330] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40: stopping tablet replica
I20260620 21:45:53.828758 9061 raft_consensus.cc:2229] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:53.829129 9061 raft_consensus.cc:2258] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:53.833281 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.816974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48406 (local address 127.1.155.189:38729)
0620 21:45:53.817308 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.817313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.817327 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:45:53.817366 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.817368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.817414 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.817496 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.825058 (+ 7562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.825942 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.827065 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.827420 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.827527 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.829804 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.829808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.829810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.829867 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.831354 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.831768 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.831773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.831775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.831859 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.832168 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.832172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.832174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.832391 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.832532 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.832966 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.833088 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266}
I20260620 21:45:53.834955 9061 ts_tablet_manager.cc:1905] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:53.837698 8736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.827321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46328 (local address 127.1.155.188:40219)
0620 21:45:53.827436 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.827441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.828257 (+ 816us) server_negotiation.cc:407] Connection header received
0620 21:45:53.828504 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.828507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.828563 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.828635 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.829595 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.830566 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.831861 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.832167 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.832276 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.833772 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.833776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.833778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.833804 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.835704 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.836193 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.836196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.836198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.836288 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.836619 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.836623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.836625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.836869 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.836979 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.837424 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.837537 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:45:53.838171 9061 ts_tablet_manager.cc:1918] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:45:53.838281 9061 log.cc:1196] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/d0ff5195dd8e4c0187a3704f0720d59a
I20260620 21:45:53.841243 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.816491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.1.155.188:40219)
0620 21:45:53.816623 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.816626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.833237 (+ 16611us) server_negotiation.cc:407] Connection header received
0620 21:45:53.833417 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.833423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.833468 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.833514 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.834270 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.834893 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.835896 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.836222 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.836326 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.837933 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.837936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.837939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.837961 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.839560 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.840033 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.840036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.840038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.840126 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.840398 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.840401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.840403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.840558 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.840690 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.840944 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.841058 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:45:53.846530 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:53.828083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48408 (local address 127.1.155.189:38729)
0620 21:45:53.828208 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:53.828215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:53.838301 (+ 10086us) server_negotiation.cc:407] Connection header received
0620 21:45:53.838518 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:53.838521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:53.838567 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:53.838621 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:53.839438 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.840061 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.841267 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:53.841640 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:53.841716 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:53.843435 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:53.843437 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:53.843439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:53.843459 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:53.844815 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.845284 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.845286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.845287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.845378 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:53.845661 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:53.845664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:53.845665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:53.845823 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:53.845983 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:53.846226 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 21:45:53.846346 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:45:53.850147 9061 ts_tablet_manager.cc:1939] T d0ff5195dd8e4c0187a3704f0720d59a P ab51c4cca72e4d74a82e11db2a47fc40: Deleting consensus metadata
W20260620 21:45:53.851305 8756 connection.cc:441] server connection from 127.0.0.1:58206 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58206 (request call id 1) could send its response
I20260620 21:45:54.802235 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:54.801702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.131:47603)
0620 21:45:54.801997 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:54.802107 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":134,"threads_started":1}
I20260620 21:45:54.802732 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:54.802269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.129:38753)
0620 21:45:54.802545 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:54.802640 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":67,"threads_started":1}
W20260620 21:45:54.802737 9004 heartbeater.cc:644] Failed to heartbeat to 127.1.155.190:36873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
W20260620 21:45:54.803023 8730 heartbeater.cc:644] Failed to heartbeat to 127.1.155.190:36873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
I20260620 21:45:54.803606 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:54.803260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.130:46119)
0620 21:45:54.803467 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:54.803532 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
W20260620 21:45:54.803972 8867 heartbeater.cc:644] Failed to heartbeat to 127.1.155.190:36873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
I20260620 21:45:55.315269 9074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32c4b8ae8381486abded8f30d55598ee)
I20260620 21:45:55.315455 9074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } }
I20260620 21:45:55.315689 9075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32c4b8ae8381486abded8f30d55598ee)
I20260620 21:45:55.315838 9075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } }
I20260620 21:45:55.316406 9075 leader_election.cc:288] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32c4b8ae8381486abded8f30d55598ee (127.1.155.190:36873), 0c67cbaf33094ee3a55bb073c12e4749 (127.1.155.189:38729)
I20260620 21:45:55.316684 9074 leader_election.cc:288] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32c4b8ae8381486abded8f30d55598ee (127.1.155.190:36873), 0ec4a9136bbd4d87ac89b1d9a9b104db (127.1.155.188:40219)
I20260620 21:45:55.317108 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.316562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.0.0.1:49204)
0620 21:45:55.316909 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.317002 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":72,"threads_started":1}
W20260620 21:45:55.317329 8536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
I20260620 21:45:55.317569 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.316810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.0.0.1:49210)
0620 21:45:55.317326 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.317461 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":111,"threads_started":1}
W20260620 21:45:55.317909 8346 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
I20260620 21:45:55.318572 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.318347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.0.0.1:49224)
0620 21:45:55.318471 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.318512 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 21:45:55.318910 8536 leader_election.cc:334] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 32c4b8ae8381486abded8f30d55598ee (127.1.155.190:36873): Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
I20260620 21:45:55.319041 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.318794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.0.0.1:49232)
0620 21:45:55.318918 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.318975 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 21:45:55.319164 8346 leader_election.cc:334] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 32c4b8ae8381486abded8f30d55598ee (127.1.155.190:36873): Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
I20260620 21:45:55.321327 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.316622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38729 (local address 127.0.0.1:41190)
0620 21:45:55.316909 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.316929 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:55.317045 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:55.317611 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:55.317615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:55.317634 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:55.317747 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:55.317775 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.318912 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.318915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:55.320190 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:55.320200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.320949 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.320952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:55.321019 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:55.321044 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:45:55.321108 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 21:45:55.321236 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":44,"thread_start_us":61,"threads_started":1}
I20260620 21:45:55.321331 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.316806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.1.155.189:38729)
0620 21:45:55.317341 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:55.317347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:55.317364 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:45:55.317410 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:55.317415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:55.317459 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:55.317903 (+ 444us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:55.318006 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.318767 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.320328 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.320827 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.320951 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:55.321006 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:55.321140 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 21:45:55.321214 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":52,"threads_started":1}
I20260620 21:45:55.321741 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0c67cbaf33094ee3a55bb073c12e4749" is_pre_election: true
I20260620 21:45:55.321959 8370 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ec4a9136bbd4d87ac89b1d9a9b104db in term 1.
I20260620 21:45:55.322063 9081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.317410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40218 (local address 127.1.155.188:40219)
0620 21:45:55.317971 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:55.317976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:55.317991 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:45:55.318042 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:55.318045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:55.318092 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:55.318500 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:55.318607 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.319383 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.320693 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.321382 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.321595 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:55.321653 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:55.321865 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 21:45:55.321939 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":463,"thread_start_us":67,"threads_started":1}
I20260620 21:45:55.322074 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.317059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:40219 (local address 127.0.0.1:40218)
0620 21:45:55.317616 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.317633 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:55.317728 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:55.318232 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:55.318236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:55.318259 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:55.318365 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:55.318385 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.319494 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.319498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:55.320563 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:55.320571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.321587 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.321591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:55.321681 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:55.321707 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 21:45:55.321791 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 21:45:55.321897 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":60,"threads_started":1}
I20260620 21:45:55.322355 8537 leader_election.cc:302] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c67cbaf33094ee3a55bb073c12e4749, 0ec4a9136bbd4d87ac89b1d9a9b104db; no voters: 32c4b8ae8381486abded8f30d55598ee
I20260620 21:45:55.322542 9075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 21:45:55.322603 9075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32c4b8ae8381486abded8f30d55598ee)
I20260620 21:45:55.322646 9075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:45:55.322835 8560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c67cbaf33094ee3a55bb073c12e4749" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" is_pre_election: true
I20260620 21:45:55.323704 9075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } }
I20260620 21:45:55.323979 8560 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c67cbaf33094ee3a55bb073c12e4749 in current term 2: Already voted for candidate 0ec4a9136bbd4d87ac89b1d9a9b104db in this term.
I20260620 21:45:55.324095 9075 leader_election.cc:288] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [CANDIDATE]: Term 2 election: Requested vote from peers 32c4b8ae8381486abded8f30d55598ee (127.1.155.190:36873), 0c67cbaf33094ee3a55bb073c12e4749 (127.1.155.189:38729)
I20260620 21:45:55.324359 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0c67cbaf33094ee3a55bb073c12e4749"
I20260620 21:45:55.324501 8370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:45:55.324719 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.324213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.0.0.1:49234)
0620 21:45:55.324593 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.324641 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317}
I20260620 21:45:55.324748 8345 leader_election.cc:302] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c67cbaf33094ee3a55bb073c12e4749; no voters: 0ec4a9136bbd4d87ac89b1d9a9b104db, 32c4b8ae8381486abded8f30d55598ee
I20260620 21:45:55.325333 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.325130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.0.0.1:49238)
0620 21:45:55.325228 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.325275 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 21:45:55.325443 8536 leader_election.cc:334] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 32c4b8ae8381486abded8f30d55598ee (127.1.155.190:36873): Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
I20260620 21:45:55.325636 8370 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec4a9136bbd4d87ac89b1d9a9b104db in term 2.
I20260620 21:45:55.325801 9074 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:45:55.325927 8537 leader_election.cc:302] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c67cbaf33094ee3a55bb073c12e4749, 0ec4a9136bbd4d87ac89b1d9a9b104db; no voters: 32c4b8ae8381486abded8f30d55598ee
I20260620 21:45:55.326136 9075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:45:55.326329 9075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [term 2 LEADER]: Becoming Leader. State: Replica: 0ec4a9136bbd4d87ac89b1d9a9b104db, State: Running, Role: LEADER
I20260620 21:45:55.326577 9075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } }
I20260620 21:45:55.327574 9084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ec4a9136bbd4d87ac89b1d9a9b104db. Latest consensus state: current_term: 2 leader_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } } }
I20260620 21:45:55.327662 9084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:55.327946 9086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:55.330331 9086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:55.330750 9086 catalog_manager.cc:1259] Loaded cluster ID: 4a3ea318e1ba4c3598f9fdf369f79031
I20260620 21:45:55.330807 9086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:55.331341 9086 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:55.331740 9086 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: Loaded TSK: 0
I20260620 21:45:55.332401 9086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:45:55.367586 8550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:45:55.371033 8550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:55.374131 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.373801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.0.0.1:49248)
0620 21:45:55.374001 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.374059 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260620 21:45:55.374189 8370 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [term 2 FOLLOWER]: Refusing update from remote peer 0ec4a9136bbd4d87ac89b1d9a9b104db: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 21:45:55.374784 9075 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 21:45:55.374925 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.374699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.0.0.1:49250)
0620 21:45:55.374815 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.374868 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 21:45:55.375037 8536 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db -> Peer 32c4b8ae8381486abded8f30d55598ee (127.1.155.190:36873): Couldn't send request to peer 32c4b8ae8381486abded8f30d55598ee. Status: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 21:45:55.376194 9074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ec4a9136bbd4d87ac89b1d9a9b104db. Latest consensus state: current_term: 2 leader_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } } }
I20260620 21:45:55.376348 9074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:45:55.377285 9084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } } }
I20260620 21:45:55.377411 9084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:55.377626 9074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "32c4b8ae8381486abded8f30d55598ee" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36873 } } peers { permanent_uuid: "0c67cbaf33094ee3a55bb073c12e4749" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38729 } attrs { promote: false } } peers { permanent_uuid: "0ec4a9136bbd4d87ac89b1d9a9b104db" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 40219 } attrs { promote: false } } }
I20260620 21:45:55.377748 9074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c67cbaf33094ee3a55bb073c12e4749 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:45:55.388060 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.383743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.1.155.129:41139)
0620 21:45:55.384083 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:55.384087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:55.384098 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:45:55.384139 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:55.384141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:55.384205 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:55.384596 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:55.384709 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.385450 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.386880 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.387497 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.387614 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:55.387680 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:55.387851 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 21:45:55.387925 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":74,"threads_started":1}
I20260620 21:45:55.388054 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.384208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:36809 (local address 127.0.0.1:49942)
0620 21:45:55.384289 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.384305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:55.384379 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:55.384958 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:55.384961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:55.385001 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:55.385075 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:55.385080 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.386037 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.386044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:55.386800 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:55.386809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.387668 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.387671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:55.387740 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:55.387757 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 21:45:55.387800 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 21:45:55.387895 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":4,"spinlock_wait_cycles":768}
I20260620 21:45:55.388053 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.383632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:41139 (local address 127.0.0.1:35250)
0620 21:45:55.383771 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.383787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:55.383876 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:55.384326 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:55.384329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:55.384345 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:55.384488 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:55.384497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.385568 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.385571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:55.386747 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:55.386755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.387627 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.387632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:55.387723 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:55.387748 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:45:55.387825 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 21:45:55.387904 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":21,"spinlock_wait_cycles":1280}
I20260620 21:45:55.388646 8663 tablet_service.cc:1467] Processing CreateTablet for tablet 24561a28aee14dbf867160944faead57 (DEFAULT_TABLE table=test-table [id=a62fe978574d4275a35f02c71a00192a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:55.388929 8663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24561a28aee14dbf867160944faead57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:55.389011 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.384331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.1.155.130:36809)
0620 21:45:55.384729 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:55.384734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:55.384747 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:45:55.384791 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:55.384794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:55.384837 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:55.385175 (+ 338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:55.385272 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.385927 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.386941 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.387531 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.388688 (+ 1157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:55.388763 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:55.388803 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 21:45:55.388873 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":67,"threads_started":1}
I20260620 21:45:55.389467 8800 tablet_service.cc:1467] Processing CreateTablet for tablet 24561a28aee14dbf867160944faead57 (DEFAULT_TABLE table=test-table [id=a62fe978574d4275a35f02c71a00192a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:55.389703 8800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24561a28aee14dbf867160944faead57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:55.391577 9097 tablet_bootstrap.cc:492] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967: Bootstrap starting.
I20260620 21:45:55.391880 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.384403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:41081 (local address 127.0.0.1:59906)
0620 21:45:55.384889 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.384902 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:55.384983 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:55.385210 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:55.385213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:55.385227 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:55.385321 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:55.385327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.386383 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.386386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:55.387473 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:55.387483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.388387 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.388390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:55.391561 (+ 3171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:55.391586 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:45:55.391668 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 21:45:55.391742 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":110,"threads_started":1}
I20260620 21:45:55.392269 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.384745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59906 (local address 127.1.155.131:41081)
0620 21:45:55.384983 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:55.384987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:55.385014 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:45:55.385067 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:55.385070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:55.385113 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:55.385204 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:55.385431 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.386259 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.387636 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.388260 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.388390 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:55.388449 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:55.392037 (+ 3588us) server_negotiation.cc:299] Negotiation successful
0620 21:45:55.392129 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
I20260620 21:45:55.392304 9098 tablet_bootstrap.cc:492] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40: Bootstrap starting.
I20260620 21:45:55.392777 9097 tablet_bootstrap.cc:654] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:55.392731 8937 tablet_service.cc:1467] Processing CreateTablet for tablet 24561a28aee14dbf867160944faead57 (DEFAULT_TABLE table=test-table [id=a62fe978574d4275a35f02c71a00192a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:55.392968 8937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24561a28aee14dbf867160944faead57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:55.393349 9098 tablet_bootstrap.cc:654] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:55.394018 9097 tablet_bootstrap.cc:492] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967: No bootstrap required, opened a new log
I20260620 21:45:55.394202 9097 ts_tablet_manager.cc:1397] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:45:55.394541 9098 tablet_bootstrap.cc:492] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40: No bootstrap required, opened a new log
I20260620 21:45:55.394660 9098 ts_tablet_manager.cc:1397] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:45:55.394693 9097 raft_consensus.cc:348] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } }
I20260620 21:45:55.394810 9097 raft_consensus.cc:374] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:55.394846 9097 raft_consensus.cc:729] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a89e130a404ae68c475898d6f89967, State: Initialized, Role: FOLLOWER
I20260620 21:45:55.394980 9097 consensus_queue.cc:260] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } }
I20260620 21:45:55.395092 9098 raft_consensus.cc:348] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } }
I20260620 21:45:55.395227 9098 raft_consensus.cc:374] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:55.395265 9098 raft_consensus.cc:729] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab51c4cca72e4d74a82e11db2a47fc40, State: Initialized, Role: FOLLOWER
I20260620 21:45:55.395385 9097 ts_tablet_manager.cc:1428] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:55.395372 9098 consensus_queue.cc:260] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } }
I20260620 21:45:55.395700 9098 ts_tablet_manager.cc:1428] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:55.397851 8729 heartbeater.cc:503] Master 127.1.155.188:40219 was elected leader, sending a full tablet report...
I20260620 21:45:55.397917 9102 tablet_bootstrap.cc:492] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4: Bootstrap starting.
I20260620 21:45:55.398280 8866 heartbeater.cc:503] Master 127.1.155.188:40219 was elected leader, sending a full tablet report...
I20260620 21:45:55.399560 9102 tablet_bootstrap.cc:654] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:55.399768 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.397324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.130:44239)
0620 21:45:55.399489 (+ 2165us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.399659 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2078,"thread_start_us":96,"threads_started":1}
I20260620 21:45:55.400163 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.395870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.129:40275)
0620 21:45:55.399983 (+ 4113us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.400050 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4051,"thread_start_us":88,"threads_started":1}
I20260620 21:45:55.400605 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.400199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.130:56321)
0620 21:45:55.400506 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.400557 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240}
I20260620 21:45:55.400936 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.400605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.129:49993)
0620 21:45:55.400828 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.400882 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
I20260620 21:45:55.400942 9102 tablet_bootstrap.cc:492] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4: No bootstrap required, opened a new log
I20260620 21:45:55.401046 9102 ts_tablet_manager.cc:1397] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:45:55.401429 9102 raft_consensus.cc:348] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } }
I20260620 21:45:55.401546 9102 raft_consensus.cc:374] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:55.401589 9102 raft_consensus.cc:729] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231e4941b8d34841bd74c6b0aea79ab4, State: Initialized, Role: FOLLOWER
I20260620 21:45:55.401726 9102 consensus_queue.cc:260] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } }
I20260620 21:45:55.402045 9102 ts_tablet_manager.cc:1428] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:55.402774 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.402404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.131:40409)
0620 21:45:55.402627 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.402685 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260620 21:45:55.403517 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.403161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.131:45793)
0620 21:45:55.403376 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.403426 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
I20260620 21:45:55.403607 9003 heartbeater.cc:503] Master 127.1.155.188:40219 was elected leader, sending a full tablet report...
I20260620 21:45:55.452072 9101 raft_consensus.cc:482] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:55.452196 9101 raft_consensus.cc:504] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } }
I20260620 21:45:55.452543 9101 leader_election.cc:288] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab51c4cca72e4d74a82e11db2a47fc40 (127.1.155.130:36809), 231e4941b8d34841bd74c6b0aea79ab4 (127.1.155.131:41081)
I20260620 21:45:55.453080 8820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24561a28aee14dbf867160944faead57" candidate_uuid: "b2a89e130a404ae68c475898d6f89967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" is_pre_election: true
I20260620 21:45:55.453083 8957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24561a28aee14dbf867160944faead57" candidate_uuid: "b2a89e130a404ae68c475898d6f89967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231e4941b8d34841bd74c6b0aea79ab4" is_pre_election: true
I20260620 21:45:55.453215 8820 raft_consensus.cc:2454] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a89e130a404ae68c475898d6f89967 in term 0.
I20260620 21:45:55.453262 8957 raft_consensus.cc:2454] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a89e130a404ae68c475898d6f89967 in term 0.
I20260620 21:45:55.453608 8618 leader_election.cc:302] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231e4941b8d34841bd74c6b0aea79ab4, b2a89e130a404ae68c475898d6f89967; no voters:
I20260620 21:45:55.453826 9101 raft_consensus.cc:2781] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:55.453950 9101 raft_consensus.cc:482] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:55.454035 9101 raft_consensus.cc:3037] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:55.455056 9101 raft_consensus.cc:504] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } }
I20260620 21:45:55.455385 9101 leader_election.cc:288] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [CANDIDATE]: Term 1 election: Requested vote from peers ab51c4cca72e4d74a82e11db2a47fc40 (127.1.155.130:36809), 231e4941b8d34841bd74c6b0aea79ab4 (127.1.155.131:41081)
I20260620 21:45:55.455825 8957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24561a28aee14dbf867160944faead57" candidate_uuid: "b2a89e130a404ae68c475898d6f89967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231e4941b8d34841bd74c6b0aea79ab4"
I20260620 21:45:55.455825 8820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24561a28aee14dbf867160944faead57" candidate_uuid: "b2a89e130a404ae68c475898d6f89967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab51c4cca72e4d74a82e11db2a47fc40"
I20260620 21:45:55.455955 8957 raft_consensus.cc:3037] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:55.455981 8820 raft_consensus.cc:3037] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:55.457078 8957 raft_consensus.cc:2454] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a89e130a404ae68c475898d6f89967 in term 1.
I20260620 21:45:55.457078 8820 raft_consensus.cc:2454] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a89e130a404ae68c475898d6f89967 in term 1.
I20260620 21:45:55.457392 8618 leader_election.cc:302] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231e4941b8d34841bd74c6b0aea79ab4, b2a89e130a404ae68c475898d6f89967; no voters:
I20260620 21:45:55.457576 9101 raft_consensus.cc:2781] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:55.457880 9101 raft_consensus.cc:686] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [term 1 LEADER]: Becoming Leader. State: Replica: b2a89e130a404ae68c475898d6f89967, State: Running, Role: LEADER
I20260620 21:45:55.458052 9101 consensus_queue.cc:237] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } } peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } }
I20260620 21:45:55.458730 9107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.458407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.129:60605)
0620 21:45:55.458557 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.458623 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 21:45:55.460302 8549 catalog_manager.cc:5495] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2a89e130a404ae68c475898d6f89967 (127.1.155.129). New cstate: current_term: 1 leader_uuid: "b2a89e130a404ae68c475898d6f89967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a89e130a404ae68c475898d6f89967" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 } health_report { overall_health: UNKNOWN } } }
I20260620 21:45:55.486850 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.483710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.1.155.129:41139)
0620 21:45:55.483868 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:55.483875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:55.483901 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:45:55.483989 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:55.483991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:55.484033 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:55.484123 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:55.484365 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.484899 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.485751 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:55.486013 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:55.486125 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:55.486357 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:55.486433 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:55.486517 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:55.486640 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 21:45:55.486691 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 21:45:55.489039 8957 raft_consensus.cc:1261] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4 [term 1 FOLLOWER]: Refusing update from remote peer b2a89e130a404ae68c475898d6f89967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:55.489136 8820 raft_consensus.cc:1261] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40 [term 1 FOLLOWER]: Refusing update from remote peer b2a89e130a404ae68c475898d6f89967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:55.489656 9101 consensus_queue.cc:1035] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab51c4cca72e4d74a82e11db2a47fc40" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 36809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:55.489815 9111 consensus_queue.cc:1035] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231e4941b8d34841bd74c6b0aea79ab4" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:55.491623 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.491360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.130:42083)
0620 21:45:55.491466 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.491534 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 21:45:55.492620 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.492348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.131:58985)
0620 21:45:55.492494 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.492550 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 21:45:55.494179 9114 mvcc.cc:205] Tried to move back new op lower bound from 7299039049678696448 to 7299039049559412736. Current Snapshot: MvccSnapshot[applied={T|T < 7299039049678696448}]
I20260620 21:45:55.495215 9107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.494956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.129:60969)
0620 21:45:55.495090 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.495147 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 21:45:55.496402 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.496123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.131:52305)
0620 21:45:55.496269 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.496322 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 21:45:55.497526 9107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.497268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.129:52409)
0620 21:45:55.497393 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.497455 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 21:45:55.498119 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.496019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36873 (local address 127.1.155.130:48347)
0620 21:45:55.497996 (+ 1977us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.498051 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1910}
I20260620 21:45:55.496757 9116 mvcc.cc:205] Tried to move back new op lower bound from 7299039049678696448 to 7299039049559412736. Current Snapshot: MvccSnapshot[applied={T|T < 7299039049678696448}]
I20260620 21:45:55.499603 8548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:55.499751 8548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:55.504125 8548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ec4a9136bbd4d87ac89b1d9a9b104db: Sending DeleteTablet for 3 replicas of tablet 24561a28aee14dbf867160944faead57
I20260620 21:45:55.504647 8663 tablet_service.cc:1514] Processing DeleteTablet for tablet 24561a28aee14dbf867160944faead57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:55 UTC) from {username='oryx', principal='oryx/127.1.155.188@KRBTEST.COM'} at 127.0.0.1:35250
I20260620 21:45:55.504647 8800 tablet_service.cc:1514] Processing DeleteTablet for tablet 24561a28aee14dbf867160944faead57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:55 UTC) from {username='oryx', principal='oryx/127.1.155.188@KRBTEST.COM'} at 127.0.0.1:49942
I20260620 21:45:55.504942 8799 tablet_service.cc:1514] Processing DeleteTablet for tablet 24561a28aee14dbf867160944faead57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:55 UTC) from {username='oryx', principal='oryx/127.1.155.188@KRBTEST.COM'} at 127.0.0.1:49942
I20260620 21:45:55.504979 9123 tablet_replica.cc:330] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967: stopping tablet replica
I20260620 21:45:55.505096 9123 raft_consensus.cc:2229] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:45:55.505225 8937 tablet_service.cc:1514] Processing DeleteTablet for tablet 24561a28aee14dbf867160944faead57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:55 UTC) from {username='oryx', principal='oryx/127.1.155.188@KRBTEST.COM'} at 127.0.0.1:59906
I20260620 21:45:55.505277 9123 raft_consensus.cc:2258] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:55.505295 9122 tablet_replica.cc:330] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40: stopping tablet replica
I20260620 21:45:55.505585 9124 tablet_replica.cc:330] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4: stopping tablet replica
I20260620 21:45:55.505623 9122 raft_consensus.cc:2229] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:55.505708 9124 raft_consensus.cc:2229] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:55.505856 9124 raft_consensus.cc:2258] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:55.505877 9122 raft_consensus.cc:2258] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:55.506102 9123 ts_tablet_manager.cc:1905] T 24561a28aee14dbf867160944faead57 P b2a89e130a404ae68c475898d6f89967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:55.506639 9122 ts_tablet_manager.cc:1905] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:55.506805 9124 ts_tablet_manager.cc:1905] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:55.506968 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 8505
I20260620 21:45:55.509944 9122 ts_tablet_manager.cc:1918] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:45:55.510093 9122 log.cc:1196] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/24561a28aee14dbf867160944faead57
I20260620 21:45:55.510208 9124 ts_tablet_manager.cc:1918] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:45:55.510286 9124 log.cc:1196] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/24561a28aee14dbf867160944faead57
I20260620 21:45:55.510560 9122 ts_tablet_manager.cc:1939] T 24561a28aee14dbf867160944faead57 P ab51c4cca72e4d74a82e11db2a47fc40: Deleting consensus metadata
I20260620 21:45:55.510560 9124 ts_tablet_manager.cc:1939] T 24561a28aee14dbf867160944faead57 P 231e4941b8d34841bd74c6b0aea79ab4: Deleting consensus metadata
I20260620 21:45:55.511240 8536 catalog_manager.cc:4841] TS 231e4941b8d34841bd74c6b0aea79ab4 (127.1.155.131:41081): tablet 24561a28aee14dbf867160944faead57 (table test-table [id=a62fe978574d4275a35f02c71a00192a]) successfully deleted
I20260620 21:45:55.511432 8536 catalog_manager.cc:4841] TS ab51c4cca72e4d74a82e11db2a47fc40 (127.1.155.130:36809): tablet 24561a28aee14dbf867160944faead57 (table test-table [id=a62fe978574d4275a35f02c71a00192a]) successfully deleted
W20260620 21:45:55.511734 8536 catalog_manager.cc:4808] TS ab51c4cca72e4d74a82e11db2a47fc40 (127.1.155.130:36809): delete failed for tablet 24561a28aee14dbf867160944faead57 because the tablet was not found. No further retry: Not found: Tablet not found: 24561a28aee14dbf867160944faead57
I20260620 21:45:55.517601 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 8737
I20260620 21:45:55.517795 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:55.517503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:41139 (local address 127.0.0.1:35270)
0620 21:45:55.517638 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:55.517699 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:41139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 21:45:55.518054 8535 catalog_manager.cc:4584] TS b2a89e130a404ae68c475898d6f89967 (127.1.155.129:41139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24561a28aee14dbf867160944faead57: Network error: Client connection negotiation failed: client connection to 127.1.155.129:41139: connect: Connection refused (error 111)
I20260620 21:45:55.523528 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 8873
I20260620 21:45:55.529160 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 8222
I20260620 21:45:55.535475 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 8324
2026-06-20T21:45:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5453 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:45:55 dist-test-slave-5bvn krb5kdc[9127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:45:55 dist-test-slave-5bvn krb5kdc[9127](info): set up 2 sockets
Jun 20 21:45:55 dist-test-slave-5bvn krb5kdc[9127](info): commencing operation
krb5kdc: starting...
W20260620 21:45:56.586485 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:45:56 dist-test-slave-5bvn krb5kdc[9127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:45:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:45:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.190.keytab.
Entry for principal oryx/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.190.keytab.
I20260620 21:45:56.709079 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:34253
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.190.keytab
--principal=oryx/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:46131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:34253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:56.816334 9140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:56.816622 9140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:56.816695 9140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:56.820768 9140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:45:56.820851 9140 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 21:45:56.820883 9140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:56.820926 9140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:56.820960 9140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:45:56.820995 9140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:45:56.826723 9140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:46131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:34253
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.190.keytab
--principal=oryx/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:34253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:56.827719 9140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:56.828665 9140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:56.837134 9146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:56.837141 9145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:56.837209 9148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:56.837381 9140 server_base.cc:1034] running on GCE node
I20260620 21:45:56.838079 9140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:56.839149 9140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:56.840334 9140 hybrid_clock.cc:648] HybridClock initialized: now 1781991956840320 us; error 27 us; skew 500 ppm
Jun 20 21:45:56 dist-test-slave-5bvn krb5kdc[9127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991956, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:56.843179 9140 init.cc:377] Logged in from keytab as oryx/127.1.155.190@KRBTEST.COM (short username oryx)
I20260620 21:45:56.844156 9140 webserver.cc:460] Webserver started at http://127.1.155.190:44869/ using document root <none> and password file <none>
I20260620 21:45:56.844741 9140 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:56.844844 9140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:56.845077 9140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:56.847179 9140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "389ae4b24e25449d8ea659d032e2d5eb"
format_stamp: "Formatted at 2026-06-20 21:45:56 on dist-test-slave-5bvn"
server_key: "4b6b287451977722178efb1d630551d9"
server_key_iv: "2d7a7a6bd23589bcb3c8387c25e055d2"
server_key_version: "encryptionkey@0"
I20260620 21:45:56.847733 9140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "389ae4b24e25449d8ea659d032e2d5eb"
format_stamp: "Formatted at 2026-06-20 21:45:56 on dist-test-slave-5bvn"
server_key: "4b6b287451977722178efb1d630551d9"
server_key_iv: "2d7a7a6bd23589bcb3c8387c25e055d2"
server_key_version: "encryptionkey@0"
I20260620 21:45:56.851409 9140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:45:56.853991 9155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:56.855422 9140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:45:56.855599 9140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "389ae4b24e25449d8ea659d032e2d5eb"
format_stamp: "Formatted at 2026-06-20 21:45:56 on dist-test-slave-5bvn"
server_key: "4b6b287451977722178efb1d630551d9"
server_key_iv: "2d7a7a6bd23589bcb3c8387c25e055d2"
server_key_version: "encryptionkey@0"
I20260620 21:45:56.855724 9140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:56.871286 9140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:56.874680 9140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:56.874920 9140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:56.883212 9140 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:34253
I20260620 21:45:56.883236 9217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:34253 every 8 connection(s)
I20260620 21:45:56.884181 9140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:45:56.884694 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 9140
I20260620 21:45:56.884855 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:45:56.885134 1646 external_mini_cluster.cc:1428] Setting key 6141025e7bbd5d083da4d137492f7bf3
I20260620 21:45:56.887202 9218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:56.893811 9218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb: Bootstrap starting.
Jun 20 21:45:56 dist-test-slave-5bvn krb5kdc[9127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
I20260620 21:45:56.896289 9218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:56.897359 9218 log.cc:824] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:56.899361 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:56.886609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52120 (local address 127.1.155.190:34253)
0620 21:45:56.887023 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:56.887038 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:56.887074 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:45:56.887732 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:56.887750 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:56.888089 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:56.888372 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:56.889251 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:56.890315 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:56.891413 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:56.891854 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:56.891974 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:56.894815 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:56.894820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:56.894832 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:56.894858 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:56.896528 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:56.896963 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:56.896974 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:56.896979 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:56.897083 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:56.897373 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:56.897377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:56.897379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:56.897875 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:56.898090 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:56.898339 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 21:45:56.898647 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":135,"threads_started":1}
I20260620 21:45:56.899395 9218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb: No bootstrap required, opened a new log
I20260620 21:45:56.902755 9218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389ae4b24e25449d8ea659d032e2d5eb" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 34253 } }
I20260620 21:45:56.903002 9218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:56.903083 9218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389ae4b24e25449d8ea659d032e2d5eb, State: Initialized, Role: FOLLOWER
I20260620 21:45:56.903496 9218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389ae4b24e25449d8ea659d032e2d5eb" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 34253 } }
I20260620 21:45:56.903646 9218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:45:56.903733 9218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:45:56.903851 9218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:56.904733 9218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389ae4b24e25449d8ea659d032e2d5eb" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 34253 } }
I20260620 21:45:56.905066 9218 leader_election.cc:302] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 389ae4b24e25449d8ea659d032e2d5eb; no voters:
I20260620 21:45:56.905383 9218 leader_election.cc:288] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:45:56.905519 9223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:56.905782 9223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [term 1 LEADER]: Becoming Leader. State: Replica: 389ae4b24e25449d8ea659d032e2d5eb, State: Running, Role: LEADER
I20260620 21:45:56.906100 9223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389ae4b24e25449d8ea659d032e2d5eb" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 34253 } }
I20260620 21:45:56.906661 9218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:45:56.907618 9225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 389ae4b24e25449d8ea659d032e2d5eb. Latest consensus state: current_term: 1 leader_uuid: "389ae4b24e25449d8ea659d032e2d5eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389ae4b24e25449d8ea659d032e2d5eb" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 34253 } } }
I20260620 21:45:56.907752 9225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:56.908066 9224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "389ae4b24e25449d8ea659d032e2d5eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389ae4b24e25449d8ea659d032e2d5eb" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 34253 } } }
I20260620 21:45:56.908126 9232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:45:56.908172 9224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb [sys.catalog]: This master's current role is: LEADER
I20260620 21:45:56.910846 9232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:45:56.915802 9232 catalog_manager.cc:1347] Generated new cluster ID: f58b73af8cbb446b8c61de2c71cb1e4a
I20260620 21:45:56.915865 9232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:45:56.922971 9232 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:45:56.923827 9232 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:45:56.930744 9232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb: Generated new TSK 0
I20260620 21:45:56.931425 9232 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.129.keytab.
Entry for principal oryx/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.129.keytab.
I20260620 21:45:56.963074 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:34253
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.129.keytab
--principal=oryx/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:46131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:57.077492 9244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:57.077754 9244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:57.077798 9244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:57.081813 9244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:57.081944 9244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:57.081986 9244 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:45:57.082046 9244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:45:57.087493 9244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:46131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.129.keytab
--principal=oryx/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.155.190:34253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:57.088583 9244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:57.089594 9244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:57.099216 9252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:57.099251 9249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:57.099216 9250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:57.099859 9244 server_base.cc:1034] running on GCE node
I20260620 21:45:57.100276 9244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:57.100911 9244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:57.102119 9244 hybrid_clock.cc:648] HybridClock initialized: now 1781991957102105 us; error 39 us; skew 500 ppm
Jun 20 21:45:57 dist-test-slave-5bvn krb5kdc[9127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991957, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:57.105170 9244 init.cc:377] Logged in from keytab as oryx/127.1.155.129@KRBTEST.COM (short username oryx)
I20260620 21:45:57.106168 9244 webserver.cc:460] Webserver started at http://127.1.155.129:34715/ using document root <none> and password file <none>
I20260620 21:45:57.106722 9244 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:57.106848 9244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:57.107100 9244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:57.109184 9244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "047e72ff6592467eae05c0cdb0e9a570"
format_stamp: "Formatted at 2026-06-20 21:45:57 on dist-test-slave-5bvn"
server_key: "561fcc38eb2a3dcb95ea7bc74624f220"
server_key_iv: "63ad60caa1ec9756beebb10b96478caf"
server_key_version: "encryptionkey@0"
I20260620 21:45:57.109719 9244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "047e72ff6592467eae05c0cdb0e9a570"
format_stamp: "Formatted at 2026-06-20 21:45:57 on dist-test-slave-5bvn"
server_key: "561fcc38eb2a3dcb95ea7bc74624f220"
server_key_iv: "63ad60caa1ec9756beebb10b96478caf"
server_key_version: "encryptionkey@0"
I20260620 21:45:57.113230 9244 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:45:57.115694 9259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:57.116931 9244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 21:45:57.117060 9244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "047e72ff6592467eae05c0cdb0e9a570"
format_stamp: "Formatted at 2026-06-20 21:45:57 on dist-test-slave-5bvn"
server_key: "561fcc38eb2a3dcb95ea7bc74624f220"
server_key_iv: "63ad60caa1ec9756beebb10b96478caf"
server_key_version: "encryptionkey@0"
I20260620 21:45:57.117172 9244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:57.135871 9244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:57.138991 9244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:57.139200 9244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:57.140693 9244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:57.140764 9244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:57.140839 9244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:57.140880 9244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:45:57 dist-test-slave-5bvn krb5kdc[9127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991957, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
I20260620 21:45:57.151683 9244 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:36033
I20260620 21:45:57.151701 9374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:36033 every 8 connection(s)
I20260620 21:45:57.152567 9244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:45:57.155174 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.141485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:39655 (local address 127.1.155.190:34253)
0620 21:45:57.141645 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.141651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.142677 (+ 1026us) server_negotiation.cc:407] Connection header received
0620 21:45:57.143985 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.143991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.144058 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.144167 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:57.145823 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.146488 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.147666 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.147975 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.148073 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.151170 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:57.151174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:57.151177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:57.151208 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:57.152991 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:57.153583 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:57.153586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:57.153588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:57.153676 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:57.154082 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:57.154085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:57.154087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:57.154283 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:57.154401 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:57.154878 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 21:45:57.155005 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:45:57.155964 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.141995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:34253 (local address 127.1.155.129:39655)
0620 21:45:57.142495 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:57.142555 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:57.143737 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:57.144396 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:57.144406 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:57.144892 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:57.145645 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.145669 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.146619 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.146623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.147495 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.147503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.148137 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.148144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.148404 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.149090 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:57.149117 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:57.150984 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:57.153132 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:57.153137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:57.153151 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:57.153472 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:57.153769 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:57.153773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:57.153775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:57.153974 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:57.154395 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:57.154402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:57.154638 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 21:45:57.154813 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 21:45:57.155125 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
I20260620 21:45:57.157128 9375 heartbeater.cc:347] Connected to a master server at 127.1.155.190:34253
I20260620 21:45:57.157277 9375 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:57.157775 9375 heartbeater.cc:511] Master 127.1.155.190:34253 requested a full tablet report, sending...
I20260620 21:45:57.159430 9171 ts_manager.cc:194] Registered new tserver with Master: 047e72ff6592467eae05c0cdb0e9a570 (127.1.155.129:36033)
I20260620 21:45:57.159751 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 9244
I20260620 21:45:57.159929 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:45:57.160279 1646 external_mini_cluster.cc:1428] Setting key 7c35e612c10017e1bfc051ed6c0ed80a
I20260620 21:45:57.160729 9171 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:39655
I20260620 21:45:57.168605 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.159297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:34253 (local address 127.1.155.129:53417)
0620 21:45:57.160768 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:57.160786 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:57.160925 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:57.161269 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:57.161272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:57.161487 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:57.161617 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.161624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.162484 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.162488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.163100 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.163108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.163706 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.163709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.163785 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.164344 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:57.164357 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:57.164674 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:57.166722 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:57.166725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:57.166727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:57.166931 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:57.167512 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:57.167514 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:57.167516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:57.167603 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:57.168146 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:57.168149 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:57.168235 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 21:45:57.168333 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 21:45:57.168453 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260620 21:45:57.168751 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.159349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:53417 (local address 127.1.155.190:34253)
0620 21:45:57.159474 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.159479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.160859 (+ 1380us) server_negotiation.cc:407] Connection header received
0620 21:45:57.161033 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.161037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.161078 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.161126 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:57.161733 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.162365 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.163216 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.163584 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.163696 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.164845 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:57.164852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:57.164858 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:57.164889 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:57.166590 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:57.167035 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:57.167041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:57.167046 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:57.167137 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:57.167714 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:57.167719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:57.167724 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:57.168005 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:57.168132 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:57.168420 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 21:45:57.168554 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":44}
WARNING: no policy specified for oryx/127.1.155.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.155.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.130.keytab.
Entry for principal oryx/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.130.keytab.
I20260620 21:45:57.189368 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:34253
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.130.keytab
--principal=oryx/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:46131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:57.300115 9384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:57.300367 9384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:57.300428 9384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:57.304399 9384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:57.304489 9384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:57.304559 9384 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:45:57.304613 9384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:45:57.310117 9384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:46131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.130.keytab
--principal=oryx/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.155.190:34253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:57.311086 9384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:57.312050 9384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:57.321341 9390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:57.321388 9392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:57.321388 9389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:57.321807 9384 server_base.cc:1034] running on GCE node
I20260620 21:45:57.322227 9384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:57.322808 9384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:57.324000 9384 hybrid_clock.cc:648] HybridClock initialized: now 1781991957323984 us; error 29 us; skew 500 ppm
Jun 20 21:45:57 dist-test-slave-5bvn krb5kdc[9127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991957, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:57.327044 9384 init.cc:377] Logged in from keytab as oryx/127.1.155.130@KRBTEST.COM (short username oryx)
I20260620 21:45:57.328023 9384 webserver.cc:460] Webserver started at http://127.1.155.130:39397/ using document root <none> and password file <none>
I20260620 21:45:57.328609 9384 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:57.328683 9384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:57.328908 9384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:57.331130 9384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9"
format_stamp: "Formatted at 2026-06-20 21:45:57 on dist-test-slave-5bvn"
server_key: "81095ad571ed31442ac8592d6842d17a"
server_key_iv: "c2affee41453c04e222dfd5c88e26f89"
server_key_version: "encryptionkey@0"
I20260620 21:45:57.331727 9384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9"
format_stamp: "Formatted at 2026-06-20 21:45:57 on dist-test-slave-5bvn"
server_key: "81095ad571ed31442ac8592d6842d17a"
server_key_iv: "c2affee41453c04e222dfd5c88e26f89"
server_key_version: "encryptionkey@0"
I20260620 21:45:57.335623 9384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 21:45:57.338145 9399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:57.339395 9384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:45:57.339493 9384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9"
format_stamp: "Formatted at 2026-06-20 21:45:57 on dist-test-slave-5bvn"
server_key: "81095ad571ed31442ac8592d6842d17a"
server_key_iv: "c2affee41453c04e222dfd5c88e26f89"
server_key_version: "encryptionkey@0"
I20260620 21:45:57.339581 9384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:57.355327 9384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:57.358659 9384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:57.358860 9384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:57.360484 9384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:57.360538 9384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:57.360600 9384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:57.360633 9384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:45:57 dist-test-slave-5bvn krb5kdc[9127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991957, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
I20260620 21:45:57.372179 9384 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:35025
I20260620 21:45:57.372231 9514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:35025 every 8 connection(s)
I20260620 21:45:57.373384 9384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:45:57.375330 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.362573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:56101 (local address 127.1.155.190:34253)
0620 21:45:57.362706 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.362711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.362859 (+ 148us) server_negotiation.cc:407] Connection header received
0620 21:45:57.363960 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.363964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.364023 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.364109 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:57.365826 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.366400 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.367699 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.367993 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.368114 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.371415 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:57.371418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:57.371421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:57.371455 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:57.373169 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:57.373666 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:57.373672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:57.373682 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:57.373788 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:57.374201 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:57.374204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:57.374205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:57.374479 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:57.374575 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:57.375101 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 21:45:57.375213 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:45:57.375754 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 9384
I20260620 21:45:57.375855 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:45:57.376107 1646 external_mini_cluster.cc:1428] Setting key ab2370ff5bc71b6e00e273074268fb50
I20260620 21:45:57.376238 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.362010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:34253 (local address 127.1.155.130:56101)
0620 21:45:57.362693 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:57.362750 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:57.363729 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:57.364337 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:57.364349 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:57.364854 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:57.365625 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.365650 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.366548 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.366551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.367551 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.367561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.368166 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.368171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.368341 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.368976 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:57.368995 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:57.371215 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:57.373297 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:57.373302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:57.373313 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:57.373565 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:57.373943 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:57.373946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:57.373948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:57.374114 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:57.374623 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:57.374629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:57.374863 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 21:45:57.375021 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 21:45:57.375276 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":154,"threads_started":1}
I20260620 21:45:57.377555 9515 heartbeater.cc:347] Connected to a master server at 127.1.155.190:34253
I20260620 21:45:57.377697 9515 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:57.378237 9515 heartbeater.cc:511] Master 127.1.155.190:34253 requested a full tablet report, sending...
I20260620 21:45:57.379339 9172 ts_manager.cc:194] Registered new tserver with Master: eebd1f41f3754be49a2b9b4c42bfcbc9 (127.1.155.130:35025)
I20260620 21:45:57.380172 9172 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:56101
WARNING: no policy specified for oryx/127.1.155.131@KRBTEST.COM; defaulting to no policy
I20260620 21:45:57.387164 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.379994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:34253 (local address 127.1.155.130:41137)
0620 21:45:57.380311 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:57.380322 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:57.380448 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:57.380789 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:57.380798 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:57.380959 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:57.381073 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.381078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.381806 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.381809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.382410 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.382417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.382877 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.382880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.382954 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.383479 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:57.383493 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:57.383819 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:57.385636 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:57.385638 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:57.385640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:57.385828 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:57.386159 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:57.386161 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:57.386163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:57.386232 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:57.386687 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:57.386689 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:57.386831 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 21:45:57.386917 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 21:45:57.387021 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":7,"thread_start_us":94,"threads_started":1}
I20260620 21:45:57.387231 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.380034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:41137 (local address 127.1.155.190:34253)
0620 21:45:57.380172 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.380177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.380391 (+ 214us) server_negotiation.cc:407] Connection header received
0620 21:45:57.380559 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.380562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.380608 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.380657 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:57.381178 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.381679 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.382523 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.382758 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.382892 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.383956 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:57.383964 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:57.383968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:57.384000 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:57.385523 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:57.385974 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:57.385977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:57.385979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:57.386065 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:57.386322 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:57.386327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:57.386330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:57.386591 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:57.386706 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:57.387004 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 21:45:57.387123 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.155.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.131.keytab.
Entry for principal oryx/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.131.keytab.
I20260620 21:45:57.405601 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:34253
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.131.keytab
--principal=oryx/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:46131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:45:57.516155 9524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:45:57.516414 9524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:45:57.516459 9524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:45:57.520924 9524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:45:57.520987 9524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:45:57.521027 9524 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:45:57.521102 9524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:45:57.526507 9524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:46131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.131.keytab
--principal=oryx/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.155.190:34253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:45:57.527522 9524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:45:57.528543 9524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:45:57.537803 9532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:57.537803 9529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:45:57.537937 9530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:45:57.538067 9524 server_base.cc:1034] running on GCE node
I20260620 21:45:57.538514 9524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:45:57.539196 9524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:45:57.540416 9524 hybrid_clock.cc:648] HybridClock initialized: now 1781991957540396 us; error 33 us; skew 500 ppm
Jun 20 21:45:57 dist-test-slave-5bvn krb5kdc[9127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991957, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:45:57.543401 9524 init.cc:377] Logged in from keytab as oryx/127.1.155.131@KRBTEST.COM (short username oryx)
I20260620 21:45:57.544257 9524 webserver.cc:460] Webserver started at http://127.1.155.131:45389/ using document root <none> and password file <none>
I20260620 21:45:57.544809 9524 fs_manager.cc:359] Metadata directory not provided
I20260620 21:45:57.544898 9524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:45:57.545125 9524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:45:57.547124 9524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "449f3cea3c4f48ef9df52e1c7a5452b8"
format_stamp: "Formatted at 2026-06-20 21:45:57 on dist-test-slave-5bvn"
server_key: "2771ba34cb246468a897d1a233c13acf"
server_key_iv: "27e4f1619b955cf76b85f44b353ba662"
server_key_version: "encryptionkey@0"
I20260620 21:45:57.547676 9524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "449f3cea3c4f48ef9df52e1c7a5452b8"
format_stamp: "Formatted at 2026-06-20 21:45:57 on dist-test-slave-5bvn"
server_key: "2771ba34cb246468a897d1a233c13acf"
server_key_iv: "27e4f1619b955cf76b85f44b353ba662"
server_key_version: "encryptionkey@0"
I20260620 21:45:57.551270 9524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:45:57.553553 9539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:57.554785 9524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:45:57.554925 9524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "449f3cea3c4f48ef9df52e1c7a5452b8"
format_stamp: "Formatted at 2026-06-20 21:45:57 on dist-test-slave-5bvn"
server_key: "2771ba34cb246468a897d1a233c13acf"
server_key_iv: "27e4f1619b955cf76b85f44b353ba662"
server_key_version: "encryptionkey@0"
I20260620 21:45:57.555037 9524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:45:57.578557 9524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:45:57.582036 9524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:45:57.582257 9524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:45:57.583678 9524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:45:57.583758 9524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:45:57.583829 9524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:45:57.583879 9524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:45:57 dist-test-slave-5bvn krb5kdc[9127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991957, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
I20260620 21:45:57.595373 9524 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:44727
I20260620 21:45:57.596397 9654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:44727 every 8 connection(s)
I20260620 21:45:57.596446 9524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:45:57.598114 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.584680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:51457 (local address 127.1.155.190:34253)
0620 21:45:57.584810 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.584813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.585967 (+ 1154us) server_negotiation.cc:407] Connection header received
0620 21:45:57.587180 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.587184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.587244 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.587345 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:57.588502 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.589113 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.590412 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.590675 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.590799 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.594026 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:57.594032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:57.594035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:57.594069 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:57.595773 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:57.596378 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:57.596382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:57.596383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:57.596476 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:57.596885 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:57.596887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:57.596889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:57.597185 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:57.597302 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:57.597781 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 21:45:57.597946 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:45:57.598842 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.585278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:34253 (local address 127.1.155.131:51457)
0620 21:45:57.585767 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:57.585808 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:57.586903 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:57.587520 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:57.587527 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:57.587866 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:57.588354 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.588367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.589244 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.589248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.590280 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.590289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.590888 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.590896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.591146 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.591893 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:57.591915 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:57.593788 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:57.595906 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:57.595912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:57.595928 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:57.596250 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:57.596591 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:57.596593 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:57.596595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:57.596752 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:57.597292 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:57.597298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:57.597534 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 21:45:57.597719 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 21:45:57.598055 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":144,"threads_started":1}
I20260620 21:45:57.599972 9655 heartbeater.cc:347] Connected to a master server at 127.1.155.190:34253
I20260620 21:45:57.600109 9655 heartbeater.cc:464] Registering TS with master...
I20260620 21:45:57.600730 9655 heartbeater.cc:511] Master 127.1.155.190:34253 requested a full tablet report, sending...
I20260620 21:45:57.601904 9171 ts_manager.cc:194] Registered new tserver with Master: 449f3cea3c4f48ef9df52e1c7a5452b8 (127.1.155.131:44727)
I20260620 21:45:57.602451 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 9524
I20260620 21:45:57.602610 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:45:57.602744 9171 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:51457
I20260620 21:45:57.602939 1646 external_mini_cluster.cc:1428] Setting key 0d5b901ee10e4e4282bdfb8819eb10e5
I20260620 21:45:57.604936 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:45:57.610980 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.602588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:34253 (local address 127.1.155.131:43507)
0620 21:45:57.603068 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:57.603084 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:57.603209 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:57.603576 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:57.603579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:57.603805 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:57.603930 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.603937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.604910 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.604912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.605529 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.605537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.606195 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.606197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.606330 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.606917 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:57.606943 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:57.607541 (+ 598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:57.609325 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:57.609330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:57.609333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:57.609635 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:57.609981 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:57.609985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:57.609989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:57.610092 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:57.610551 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:57.610553 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:57.610631 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 21:45:57.610717 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 21:45:57.610824 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":130,"threads_started":1}
I20260620 21:45:57.611195 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.602445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:43507 (local address 127.1.155.190:34253)
0620 21:45:57.602565 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.602569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.603139 (+ 570us) server_negotiation.cc:407] Connection header received
0620 21:45:57.603371 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.603374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.603447 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.603543 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:57.604035 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.604776 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.605649 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.606072 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.606203 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.607672 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:57.607675 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:57.607677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:57.607701 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:57.609208 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:57.609754 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:57.609757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:57.609758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:57.609879 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:57.610196 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:57.610198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:57.610199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:57.610435 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:57.610547 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:57.610806 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 21:45:57.610941 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 21:45:57 dist-test-slave-5bvn krb5kdc[9127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991957, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:45:57 dist-test-slave-5bvn krb5kdc[9127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:45:57 dist-test-slave-5bvn krb5kdc[9127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 21:45:57.629575 9221 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:45:57.623653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.1.155.190:34253)
0620 21:45:57.623783 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.623786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.623799 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:45:57.623833 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.623835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.623871 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.623963 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:57.624675 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.625177 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.626069 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.626286 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.626394 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.629419 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52132: BlockingRecv error: recv got EOF from 127.0.0.1:52132 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 21:45:57 dist-test-slave-5bvn krb5kdc[9127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991957, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
I20260620 21:45:57.638689 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.630973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.1.155.190:34253)
0620 21:45:57.631098 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.631102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.631218 (+ 116us) server_negotiation.cc:407] Connection header received
0620 21:45:57.631374 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.631377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.631419 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.631503 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:57.632066 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.632553 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.633317 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.633544 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.633651 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.635854 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:57.635857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:57.635860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:57.635885 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:57.637056 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:57.637461 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:57.637466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:57.637468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:57.637550 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:57.637817 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:57.637821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:57.637823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:57.638029 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:57.638165 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:57.638398 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 21:45:57.638519 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:45:57.641480 9171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:45:57.644887 9171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:57.659688 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.653393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:36033 (local address 127.0.0.1:52066)
0620 21:45:57.654528 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:57.654540 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:57.654613 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:57.655103 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:57.655108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:57.655139 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:57.655364 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.655371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.656751 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.656754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.658178 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.658183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.659256 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.659260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.659360 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.659388 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:45:57.659463 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 21:45:57.659529 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":100,"spinlock_wait_cycles":2432,"thread_start_us":92,"threads_started":1}
I20260620 21:45:57.659862 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.654022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.1.155.130:35025)
0620 21:45:57.654308 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.654318 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.654837 (+ 519us) server_negotiation.cc:407] Connection header received
0620 21:45:57.654930 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.654935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.655103 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.655235 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:57.655543 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.656261 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.658319 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.659322 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.659417 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.659554 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:57.659628 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 21:45:57.659719 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":113,"threads_started":1}
I20260620 21:45:57.660146 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.653399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.1.155.131:44727)
0620 21:45:57.654091 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.654103 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.654451 (+ 348us) server_negotiation.cc:407] Connection header received
0620 21:45:57.654531 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.654539 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.654751 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.654946 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:57.655522 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.656780 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.658038 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.658899 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.659009 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.659182 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:57.659638 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 21:45:57.659761 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":101,"threads_started":1}
I20260620 21:45:57.660629 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.654362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52066 (local address 127.1.155.129:36033)
0620 21:45:57.654662 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.654678 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.654699 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:45:57.654777 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.654785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.654965 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.655112 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:57.655499 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.656627 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.658295 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.659065 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.659163 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.659291 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:57.660383 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0620 21:45:57.660489 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
I20260620 21:45:57.659739 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.653614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:35025 (local address 127.0.0.1:33660)
0620 21:45:57.654717 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:57.654731 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:57.654822 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:57.655247 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:57.655250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:57.655263 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:57.655452 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.655459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.656469 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.656477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.658171 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.658182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.659429 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.659435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.659518 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.659537 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 21:45:57.659583 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 21:45:57.659635 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"mutex_wait_us":85,"thread_start_us":84,"threads_started":1}
I20260620 21:45:57.660307 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.653230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:44727 (local address 127.0.0.1:47418)
0620 21:45:57.654129 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:57.654180 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:57.654305 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:57.655102 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:57.655109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:57.655145 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:57.655372 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.655384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.656901 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.656904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.657923 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.657932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.659045 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.659049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.659165 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.659204 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 21:45:57.659314 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 21:45:57.659450 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":62,"spinlock_wait_cycles":7936,"thread_start_us":102,"threads_started":1}
I20260620 21:45:57.663663 9589 tablet_service.cc:1467] Processing CreateTablet for tablet f112d32657084748b0dd3571bb244e65 (DEFAULT_TABLE table=test-table [id=bb8e12227c4d47f980f8ab342d48d7a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:57.663666 9449 tablet_service.cc:1467] Processing CreateTablet for tablet f112d32657084748b0dd3571bb244e65 (DEFAULT_TABLE table=test-table [id=bb8e12227c4d47f980f8ab342d48d7a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:57.663664 9309 tablet_service.cc:1467] Processing CreateTablet for tablet f112d32657084748b0dd3571bb244e65 (DEFAULT_TABLE table=test-table [id=bb8e12227c4d47f980f8ab342d48d7a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:57.664603 9589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f112d32657084748b0dd3571bb244e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:57.664630 9309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f112d32657084748b0dd3571bb244e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:57.664630 9449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f112d32657084748b0dd3571bb244e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:57.671000 9683 tablet_bootstrap.cc:492] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8: Bootstrap starting.
I20260620 21:45:57.672392 9684 tablet_bootstrap.cc:492] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570: Bootstrap starting.
I20260620 21:45:57.672480 9685 tablet_bootstrap.cc:492] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9: Bootstrap starting.
I20260620 21:45:57.673300 9683 tablet_bootstrap.cc:654] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:57.674120 9683 log.cc:824] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:57.674474 9684 tablet_bootstrap.cc:654] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:57.674474 9685 tablet_bootstrap.cc:654] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:57.675346 9685 log.cc:824] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:57.675686 9684 log.cc:824] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570: Log is configured to *not* fsync() on all Append() calls
I20260620 21:45:57.676316 9683 tablet_bootstrap.cc:492] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8: No bootstrap required, opened a new log
I20260620 21:45:57.676508 9683 ts_tablet_manager.cc:1397] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 21:45:57.677165 9685 tablet_bootstrap.cc:492] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9: No bootstrap required, opened a new log
I20260620 21:45:57.677291 9684 tablet_bootstrap.cc:492] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570: No bootstrap required, opened a new log
I20260620 21:45:57.677332 9685 ts_tablet_manager.cc:1397] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 21:45:57.677472 9684 ts_tablet_manager.cc:1397] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:45:57.680416 9683 raft_consensus.cc:348] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } }
I20260620 21:45:57.680410 9684 raft_consensus.cc:348] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } }
I20260620 21:45:57.680663 9683 raft_consensus.cc:374] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:57.680666 9684 raft_consensus.cc:374] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:57.680720 9684 raft_consensus.cc:729] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047e72ff6592467eae05c0cdb0e9a570, State: Initialized, Role: FOLLOWER
I20260620 21:45:57.680723 9683 raft_consensus.cc:729] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449f3cea3c4f48ef9df52e1c7a5452b8, State: Initialized, Role: FOLLOWER
I20260620 21:45:57.681141 9684 consensus_queue.cc:260] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } }
I20260620 21:45:57.681151 9683 consensus_queue.cc:260] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } }
I20260620 21:45:57.681069 9685 raft_consensus.cc:348] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } }
I20260620 21:45:57.681273 9685 raft_consensus.cc:374] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:57.681324 9685 raft_consensus.cc:729] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebd1f41f3754be49a2b9b4c42bfcbc9, State: Initialized, Role: FOLLOWER
I20260620 21:45:57.681696 9685 consensus_queue.cc:260] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } }
I20260620 21:45:57.682086 9684 ts_tablet_manager.cc:1428] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 21:45:57.682129 9683 ts_tablet_manager.cc:1428] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260620 21:45:57.682415 9375 heartbeater.cc:503] Master 127.1.155.190:34253 was elected leader, sending a full tablet report...
I20260620 21:45:57.682415 9515 heartbeater.cc:503] Master 127.1.155.190:34253 was elected leader, sending a full tablet report...
I20260620 21:45:57.682542 9685 ts_tablet_manager.cc:1428] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 21:45:57.683413 9655 heartbeater.cc:503] Master 127.1.155.190:34253 was elected leader, sending a full tablet report...
I20260620 21:45:57.710999 9691 raft_consensus.cc:482] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:57.711230 9691 raft_consensus.cc:504] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } }
I20260620 21:45:57.712342 9691 leader_election.cc:288] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 449f3cea3c4f48ef9df52e1c7a5452b8 (127.1.155.131:44727), 047e72ff6592467eae05c0cdb0e9a570 (127.1.155.129:36033)
I20260620 21:45:57.716578 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.712659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:36033 (local address 127.1.155.130:36431)
0620 21:45:57.713107 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:57.713123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:57.713227 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:57.713493 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:57.713496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:57.713513 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:57.713627 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.713633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.714600 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.714602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.715425 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.715434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.716117 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.716122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.716232 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.716284 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 21:45:57.716360 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 21:45:57.716414 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":140,"thread_start_us":117,"threads_started":1}
I20260620 21:45:57.716750 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.712780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:36431 (local address 127.1.155.129:36033)
0620 21:45:57.712902 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.712907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.713208 (+ 301us) server_negotiation.cc:407] Connection header received
0620 21:45:57.713335 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.713338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.713393 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.713474 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:57.713808 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.714455 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.715557 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.715994 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.716086 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.716124 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:57.716512 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 21:45:57.716618 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:45:57.716817 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.712765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:44355 (local address 127.1.155.131:44727)
0620 21:45:57.712886 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.712890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.712907 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:45:57.712965 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.712968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.713015 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.713139 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:57.713480 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.714168 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.715681 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.716102 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.716227 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.716282 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:57.716628 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 21:45:57.716717 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:45:57.716760 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.712659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:44727 (local address 127.1.155.130:44355)
0620 21:45:57.712767 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:57.712783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:57.712875 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:57.713142 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:57.713144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:57.713161 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:57.713339 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.713346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.714277 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.714281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.715560 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.715569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.716224 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.716226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.716326 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.716474 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 21:45:57.716571 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 21:45:57.716653 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":1,"spinlock_wait_cycles":4864}
I20260620 21:45:57.717670 9609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f112d32657084748b0dd3571bb244e65" candidate_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" is_pre_election: true
I20260620 21:45:57.717669 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f112d32657084748b0dd3571bb244e65" candidate_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047e72ff6592467eae05c0cdb0e9a570" is_pre_election: true
I20260620 21:45:57.718005 9329 raft_consensus.cc:2454] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebd1f41f3754be49a2b9b4c42bfcbc9 in term 0.
I20260620 21:45:57.718012 9609 raft_consensus.cc:2454] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebd1f41f3754be49a2b9b4c42bfcbc9 in term 0.
I20260620 21:45:57.718478 9401 leader_election.cc:302] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 047e72ff6592467eae05c0cdb0e9a570, 449f3cea3c4f48ef9df52e1c7a5452b8, eebd1f41f3754be49a2b9b4c42bfcbc9; no voters:
I20260620 21:45:57.718735 9691 raft_consensus.cc:2781] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:57.718811 9691 raft_consensus.cc:482] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:57.718852 9691 raft_consensus.cc:3037] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:57.719838 9691 raft_consensus.cc:504] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } }
I20260620 21:45:57.720227 9691 leader_election.cc:288] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [CANDIDATE]: Term 1 election: Requested vote from peers 449f3cea3c4f48ef9df52e1c7a5452b8 (127.1.155.131:44727), 047e72ff6592467eae05c0cdb0e9a570 (127.1.155.129:36033)
I20260620 21:45:57.720538 9609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f112d32657084748b0dd3571bb244e65" candidate_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8"
I20260620 21:45:57.720662 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f112d32657084748b0dd3571bb244e65" candidate_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047e72ff6592467eae05c0cdb0e9a570"
I20260620 21:45:57.720657 9609 raft_consensus.cc:3037] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:57.720811 9329 raft_consensus.cc:3037] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:57.721915 9609 raft_consensus.cc:2454] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebd1f41f3754be49a2b9b4c42bfcbc9 in term 1.
I20260620 21:45:57.722038 9329 raft_consensus.cc:2454] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebd1f41f3754be49a2b9b4c42bfcbc9 in term 1.
I20260620 21:45:57.722240 9402 leader_election.cc:302] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f3cea3c4f48ef9df52e1c7a5452b8, eebd1f41f3754be49a2b9b4c42bfcbc9; no voters:
I20260620 21:45:57.722393 9691 raft_consensus.cc:2781] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:57.722608 9691 raft_consensus.cc:686] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 LEADER]: Becoming Leader. State: Replica: eebd1f41f3754be49a2b9b4c42bfcbc9, State: Running, Role: LEADER
I20260620 21:45:57.722939 9691 consensus_queue.cc:237] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } }
I20260620 21:45:57.725970 9172 catalog_manager.cc:5495] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to eebd1f41f3754be49a2b9b4c42bfcbc9 (127.1.155.130). New cstate: current_term: 1 leader_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } health_report { overall_health: HEALTHY } } }
I20260620 21:45:57.742043 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.738828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.1.155.130:35025)
0620 21:45:57.738963 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.738967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.738981 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:45:57.739051 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.739053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.739103 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.739197 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:57.739414 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.739896 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.740744 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.740958 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.741036 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.741271 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:57.741670 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:57.741755 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:57.741827 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 21:45:57.741922 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:45:57.747453 9329 raft_consensus.cc:1261] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Refusing update from remote peer eebd1f41f3754be49a2b9b4c42bfcbc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:57.747740 9609 raft_consensus.cc:1261] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Refusing update from remote peer eebd1f41f3754be49a2b9b4c42bfcbc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:57.748276 9693 consensus_queue.cc:1035] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:57.748438 9691 consensus_queue.cc:1035] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:57.754673 9696 mvcc.cc:205] Tried to move back new op lower bound from 7299039058927144960 to 7299039058836029440. Current Snapshot: MvccSnapshot[applied={T|T < 7299039058927144960}]
I20260620 21:45:57.756748 9698 mvcc.cc:205] Tried to move back new op lower bound from 7299039058927144960 to 7299039058836029440. Current Snapshot: MvccSnapshot[applied={T|T < 7299039058927144960}]
I20260620 21:45:57.763556 9171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:57.763737 9171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:57.766088 9171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb: Sending DeleteTablet for 3 replicas of tablet f112d32657084748b0dd3571bb244e65
I20260620 21:45:57.766813 9589 tablet_service.cc:1514] Processing DeleteTablet for tablet f112d32657084748b0dd3571bb244e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:57 UTC) from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:47418
I20260620 21:45:57.766876 9309 tablet_service.cc:1514] Processing DeleteTablet for tablet f112d32657084748b0dd3571bb244e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:57 UTC) from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:52066
I20260620 21:45:57.766947 9449 tablet_service.cc:1514] Processing DeleteTablet for tablet f112d32657084748b0dd3571bb244e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:57 UTC) from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:33660
I20260620 21:45:57.767495 9708 tablet_replica.cc:330] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9: stopping tablet replica
I20260620 21:45:57.767798 9708 raft_consensus.cc:2229] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:45:57.768149 9708 raft_consensus.cc:2258] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:57.768182 9171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:45:57.768582 9171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:57.769452 9708 ts_tablet_manager.cc:1905] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:57.772481 9708 ts_tablet_manager.cc:1918] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:45:57.772594 9708 log.cc:1196] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/f112d32657084748b0dd3571bb244e65
I20260620 21:45:57.772962 9708 ts_tablet_manager.cc:1939] T f112d32657084748b0dd3571bb244e65 P eebd1f41f3754be49a2b9b4c42bfcbc9: Deleting consensus metadata
I20260620 21:45:57.773440 9309 tablet_service.cc:1467] Processing CreateTablet for tablet ce6ce915163247ae92424530fb601d92 (DEFAULT_TABLE table=test-table [id=e584ccfd550f4fda9d4874d43e2829e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:57.773726 9309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6ce915163247ae92424530fb601d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:57.773808 9449 tablet_service.cc:1467] Processing CreateTablet for tablet ce6ce915163247ae92424530fb601d92 (DEFAULT_TABLE table=test-table [id=e584ccfd550f4fda9d4874d43e2829e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:57.773809 9589 tablet_service.cc:1467] Processing CreateTablet for tablet ce6ce915163247ae92424530fb601d92 (DEFAULT_TABLE table=test-table [id=e584ccfd550f4fda9d4874d43e2829e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:57.774052 9589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6ce915163247ae92424530fb601d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:57.774050 9449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6ce915163247ae92424530fb601d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:57.776316 9683 tablet_bootstrap.cc:492] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8: Bootstrap starting.
I20260620 21:45:57.776405 9685 tablet_bootstrap.cc:492] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9: Bootstrap starting.
I20260620 21:45:57.776435 9684 tablet_bootstrap.cc:492] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570: Bootstrap starting.
I20260620 21:45:57.777325 9685 tablet_bootstrap.cc:654] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:57.777325 9683 tablet_bootstrap.cc:654] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:57.777366 9684 tablet_bootstrap.cc:654] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:57.777796 9707 tablet_replica.cc:330] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570: stopping tablet replica
I20260620 21:45:57.778152 9707 raft_consensus.cc:2229] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:57.778474 9707 raft_consensus.cc:2258] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:57.778589 9706 tablet_replica.cc:330] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8: stopping tablet replica
I20260620 21:45:57.778885 9706 raft_consensus.cc:2229] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:57.779153 9706 raft_consensus.cc:2258] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:57.780176 9156 catalog_manager.cc:4841] TS eebd1f41f3754be49a2b9b4c42bfcbc9 (127.1.155.130:35025): tablet f112d32657084748b0dd3571bb244e65 (table test-table [id=bb8e12227c4d47f980f8ab342d48d7a5]) successfully deleted
I20260620 21:45:57.780563 9706 ts_tablet_manager.cc:1905] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:57.781124 9683 tablet_bootstrap.cc:492] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8: No bootstrap required, opened a new log
I20260620 21:45:57.781227 9683 ts_tablet_manager.cc:1397] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260620 21:45:57.781607 9683 raft_consensus.cc:348] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:57.781724 9683 raft_consensus.cc:374] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:57.781770 9683 raft_consensus.cc:729] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449f3cea3c4f48ef9df52e1c7a5452b8, State: Initialized, Role: FOLLOWER
I20260620 21:45:57.781942 9683 consensus_queue.cc:260] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:57.782209 9683 ts_tablet_manager.cc:1428] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:57.783140 9706 ts_tablet_manager.cc:1918] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:45:57.783190 9685 tablet_bootstrap.cc:492] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9: No bootstrap required, opened a new log
I20260620 21:45:57.783233 9706 log.cc:1196] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/f112d32657084748b0dd3571bb244e65
I20260620 21:45:57.783298 9685 ts_tablet_manager.cc:1397] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 21:45:57.783528 9706 ts_tablet_manager.cc:1939] T f112d32657084748b0dd3571bb244e65 P 449f3cea3c4f48ef9df52e1c7a5452b8: Deleting consensus metadata
I20260620 21:45:57.783816 9685 raft_consensus.cc:348] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:57.783926 9685 raft_consensus.cc:374] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:57.783958 9685 raft_consensus.cc:729] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebd1f41f3754be49a2b9b4c42bfcbc9, State: Initialized, Role: FOLLOWER
I20260620 21:45:57.784075 9685 consensus_queue.cc:260] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:57.784157 9707 ts_tablet_manager.cc:1905] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:57.784349 9685 ts_tablet_manager.cc:1428] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:57.784668 9158 catalog_manager.cc:4841] TS 449f3cea3c4f48ef9df52e1c7a5452b8 (127.1.155.131:44727): tablet f112d32657084748b0dd3571bb244e65 (table test-table [id=bb8e12227c4d47f980f8ab342d48d7a5]) successfully deleted
I20260620 21:45:57.786602 9684 tablet_bootstrap.cc:492] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570: No bootstrap required, opened a new log
I20260620 21:45:57.786705 9684 ts_tablet_manager.cc:1397] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20260620 21:45:57.786863 9707 ts_tablet_manager.cc:1918] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:45:57.786969 9707 log.cc:1196] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/wal/wals/f112d32657084748b0dd3571bb244e65
I20260620 21:45:57.787292 9707 ts_tablet_manager.cc:1939] T f112d32657084748b0dd3571bb244e65 P 047e72ff6592467eae05c0cdb0e9a570: Deleting consensus metadata
I20260620 21:45:57.787402 9684 raft_consensus.cc:348] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:57.787511 9684 raft_consensus.cc:374] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:57.787550 9684 raft_consensus.cc:729] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047e72ff6592467eae05c0cdb0e9a570, State: Initialized, Role: FOLLOWER
I20260620 21:45:57.787667 9684 consensus_queue.cc:260] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:57.787906 9684 ts_tablet_manager.cc:1428] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:45:57.788461 9157 catalog_manager.cc:4841] TS 047e72ff6592467eae05c0cdb0e9a570 (127.1.155.129:36033): tablet f112d32657084748b0dd3571bb244e65 (table test-table [id=bb8e12227c4d47f980f8ab342d48d7a5]) successfully deleted
W20260620 21:45:57.847473 9656 tablet.cc:2369] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:45:57.874975 9516 tablet.cc:2369] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:45:57.904024 9376 tablet.cc:2369] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:45:57.951349 9690 raft_consensus.cc:482] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:57.951495 9690 raft_consensus.cc:504] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:57.952661 9690 leader_election.cc:288] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 047e72ff6592467eae05c0cdb0e9a570 (127.1.155.129:36033), eebd1f41f3754be49a2b9b4c42bfcbc9 (127.1.155.130:35025)
I20260620 21:45:57.957943 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.953045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:33475 (local address 127.1.155.130:35025)
0620 21:45:57.953158 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.953165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.953254 (+ 89us) server_negotiation.cc:407] Connection header received
0620 21:45:57.953415 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.953419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.953481 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.953569 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:57.953991 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.954789 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.956584 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.957212 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.957341 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.957394 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:57.957714 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 21:45:57.957796 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:45:57.957943 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.953067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:44259 (local address 127.1.155.129:36033)
0620 21:45:57.953199 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:57.953203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:57.953389 (+ 186us) server_negotiation.cc:407] Connection header received
0620 21:45:57.953484 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:57.953487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:57.953539 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:57.953610 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:57.953991 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.954774 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.956584 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.957212 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.957346 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.957394 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:57.957714 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 21:45:57.957787 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:45:57.957963 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.952957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:35025 (local address 127.1.155.131:33475)
0620 21:45:57.953154 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:57.953170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:57.953270 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:57.953614 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:57.953617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:57.953635 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:57.953848 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.953857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.954899 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.954903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.956452 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.956463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.957344 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.957348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.957559 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.957608 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 21:45:57.957683 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 21:45:57.957780 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":242}
I20260620 21:45:57.958014 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:57.952960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:36033 (local address 127.1.155.131:44259)
0620 21:45:57.953300 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:57.953314 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:57.953403 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:57.953626 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:57.953629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:57.953640 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:57.953854 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.953859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.954899 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.954903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.956452 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:57.956463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:57.957344 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:57.957348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:57.957533 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:57.957604 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 21:45:57.957683 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 21:45:57.957771 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":259,"thread_start_us":89,"threads_started":1}
I20260620 21:45:57.958570 9469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6ce915163247ae92424530fb601d92" candidate_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" is_pre_election: true
I20260620 21:45:57.958599 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6ce915163247ae92424530fb601d92" candidate_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047e72ff6592467eae05c0cdb0e9a570" is_pre_election: true
I20260620 21:45:57.958827 9329 raft_consensus.cc:2454] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449f3cea3c4f48ef9df52e1c7a5452b8 in term 0.
I20260620 21:45:57.958863 9469 raft_consensus.cc:2454] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449f3cea3c4f48ef9df52e1c7a5452b8 in term 0.
I20260620 21:45:57.959321 9541 leader_election.cc:302] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 047e72ff6592467eae05c0cdb0e9a570, 449f3cea3c4f48ef9df52e1c7a5452b8, eebd1f41f3754be49a2b9b4c42bfcbc9; no voters:
I20260620 21:45:57.959606 9690 raft_consensus.cc:2781] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:57.959699 9690 raft_consensus.cc:482] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:57.959743 9690 raft_consensus.cc:3037] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:57.960803 9690 raft_consensus.cc:504] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:57.961150 9690 leader_election.cc:288] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [CANDIDATE]: Term 1 election: Requested vote from peers 047e72ff6592467eae05c0cdb0e9a570 (127.1.155.129:36033), eebd1f41f3754be49a2b9b4c42bfcbc9 (127.1.155.130:35025)
I20260620 21:45:57.961510 9469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6ce915163247ae92424530fb601d92" candidate_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9"
I20260620 21:45:57.961524 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6ce915163247ae92424530fb601d92" candidate_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047e72ff6592467eae05c0cdb0e9a570"
I20260620 21:45:57.961611 9469 raft_consensus.cc:3037] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:57.961643 9329 raft_consensus.cc:3037] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:57.962571 9329 raft_consensus.cc:2454] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449f3cea3c4f48ef9df52e1c7a5452b8 in term 1.
I20260620 21:45:57.962638 9469 raft_consensus.cc:2454] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449f3cea3c4f48ef9df52e1c7a5452b8 in term 1.
I20260620 21:45:57.962865 9541 leader_election.cc:302] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047e72ff6592467eae05c0cdb0e9a570, 449f3cea3c4f48ef9df52e1c7a5452b8; no voters:
I20260620 21:45:57.963039 9690 raft_consensus.cc:2781] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:57.963254 9690 raft_consensus.cc:686] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 LEADER]: Becoming Leader. State: Replica: 449f3cea3c4f48ef9df52e1c7a5452b8, State: Running, Role: LEADER
I20260620 21:45:57.963524 9690 consensus_queue.cc:237] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:57.965622 9171 catalog_manager.cc:5495] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 449f3cea3c4f48ef9df52e1c7a5452b8 (127.1.155.131). New cstate: current_term: 1 leader_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } health_report { overall_health: HEALTHY } } }
Jun 20 21:45:58 dist-test-slave-5bvn krb5kdc[9127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991956, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.190@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
I20260620 21:45:58.043843 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.035506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:34253 (local address 127.0.0.1:52164)
0620 21:45:58.035820 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:58.035832 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:58.035971 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:58.036230 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:58.036232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:58.036640 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:45:58.036740 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:58.036746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.037478 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.037484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:58.038066 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:58.038073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.038505 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.038508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:58.038586 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.039177 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:45:58.039189 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:45:58.040604 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:45:58.042300 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:58.042310 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:58.042315 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:58.042548 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:58.042865 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:45:58.042868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:45:58.042869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:45:58.042946 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:45:58.043345 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:45:58.043351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:45:58.043461 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 21:45:58.043578 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 21:45:58.043708 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":97,"threads_started":1}
I20260620 21:45:58.043985 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.035550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.1.155.190:34253)
0620 21:45:58.035723 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:58.035728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:58.035911 (+ 183us) server_negotiation.cc:407] Connection header received
0620 21:45:58.036068 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:58.036072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:58.036118 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:58.036216 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:58.036855 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.037375 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.038188 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.038397 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.038504 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.040750 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:58.040754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:58.040756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:58.040778 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:58.042187 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:58.042665 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:58.042670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:58.042676 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:58.042754 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:58.043033 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:58.043036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:58.043037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:58.043231 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:58.043310 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:58.043674 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 21:45:58.043825 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 21:45:58.047919 9171 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:52164:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 21:45:58.048563 9171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:58.053362 9309 tablet_service.cc:1467] Processing CreateTablet for tablet d6e31a67236e45dba700f43cd58db680 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aa7ccc0a4b2146e0a449e5fccbbdffba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:45:58.053362 9449 tablet_service.cc:1467] Processing CreateTablet for tablet d6e31a67236e45dba700f43cd58db680 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aa7ccc0a4b2146e0a449e5fccbbdffba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:45:58.053462 9589 tablet_service.cc:1467] Processing CreateTablet for tablet d6e31a67236e45dba700f43cd58db680 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aa7ccc0a4b2146e0a449e5fccbbdffba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:45:58.053745 9309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e31a67236e45dba700f43cd58db680. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:58.053750 9449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e31a67236e45dba700f43cd58db680. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:58.053805 9589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e31a67236e45dba700f43cd58db680. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:58.056072 9683 tablet_bootstrap.cc:492] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8: Bootstrap starting.
I20260620 21:45:58.056362 9685 tablet_bootstrap.cc:492] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9: Bootstrap starting.
I20260620 21:45:58.056625 9684 tablet_bootstrap.cc:492] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570: Bootstrap starting.
I20260620 21:45:58.056980 9683 tablet_bootstrap.cc:654] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:58.057214 9685 tablet_bootstrap.cc:654] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:58.057608 9684 tablet_bootstrap.cc:654] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:58.058470 9683 tablet_bootstrap.cc:492] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8: No bootstrap required, opened a new log
I20260620 21:45:58.058553 9683 ts_tablet_manager.cc:1397] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 21:45:58.058979 9684 tablet_bootstrap.cc:492] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570: No bootstrap required, opened a new log
I20260620 21:45:58.059028 9685 tablet_bootstrap.cc:492] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9: No bootstrap required, opened a new log
I20260620 21:45:58.059062 9684 ts_tablet_manager.cc:1397] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 21:45:58.059111 9685 ts_tablet_manager.cc:1397] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:45:58.059320 9683 raft_consensus.cc:348] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.059465 9683 raft_consensus.cc:374] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:58.059489 9683 raft_consensus.cc:729] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449f3cea3c4f48ef9df52e1c7a5452b8, State: Initialized, Role: FOLLOWER
I20260620 21:45:58.059506 9684 raft_consensus.cc:348] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.059504 9685 raft_consensus.cc:348] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.059608 9685 raft_consensus.cc:374] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:58.059608 9684 raft_consensus.cc:374] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:58.059571 9683 consensus_queue.cc:260] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.059643 9684 raft_consensus.cc:729] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047e72ff6592467eae05c0cdb0e9a570, State: Initialized, Role: FOLLOWER
I20260620 21:45:58.059666 9685 raft_consensus.cc:729] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebd1f41f3754be49a2b9b4c42bfcbc9, State: Initialized, Role: FOLLOWER
I20260620 21:45:58.059777 9685 consensus_queue.cc:260] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.059774 9684 consensus_queue.cc:260] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.059783 9690 tablet_replica.cc:439] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } }
I20260620 21:45:58.059861 9683 ts_tablet_manager.cc:1428] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260620 21:45:58.059895 9690 tablet_replica.cc:442] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:58.060045 9685 ts_tablet_manager.cc:1428] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:58.060194 9689 tablet_replica.cc:439] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } }
I20260620 21:45:58.060348 9689 tablet_replica.cc:442] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:58.060406 9703 tablet_replica.cc:439] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } }
I20260620 21:45:58.060554 9703 tablet_replica.cc:442] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:58.060829 9684 ts_tablet_manager.cc:1428] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 21:45:58.334057 9690 raft_consensus.cc:482] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:58.334194 9690 raft_consensus.cc:504] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.334599 9690 leader_election.cc:288] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 047e72ff6592467eae05c0cdb0e9a570 (127.1.155.129:36033), eebd1f41f3754be49a2b9b4c42bfcbc9 (127.1.155.130:35025)
I20260620 21:45:58.335121 9469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e31a67236e45dba700f43cd58db680" candidate_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" is_pre_election: true
I20260620 21:45:58.335132 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e31a67236e45dba700f43cd58db680" candidate_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047e72ff6592467eae05c0cdb0e9a570" is_pre_election: true
I20260620 21:45:58.335286 9469 raft_consensus.cc:2454] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449f3cea3c4f48ef9df52e1c7a5452b8 in term 0.
I20260620 21:45:58.335286 9329 raft_consensus.cc:2454] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449f3cea3c4f48ef9df52e1c7a5452b8 in term 0.
I20260620 21:45:58.335616 9540 leader_election.cc:302] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f3cea3c4f48ef9df52e1c7a5452b8, eebd1f41f3754be49a2b9b4c42bfcbc9; no voters:
I20260620 21:45:58.335826 9690 raft_consensus.cc:2781] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:58.335901 9690 raft_consensus.cc:482] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:58.335951 9690 raft_consensus.cc:3037] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:58.336884 9690 raft_consensus.cc:504] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.337236 9690 leader_election.cc:288] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [CANDIDATE]: Term 1 election: Requested vote from peers 047e72ff6592467eae05c0cdb0e9a570 (127.1.155.129:36033), eebd1f41f3754be49a2b9b4c42bfcbc9 (127.1.155.130:35025)
I20260620 21:45:58.337575 9469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e31a67236e45dba700f43cd58db680" candidate_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9"
I20260620 21:45:58.337625 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e31a67236e45dba700f43cd58db680" candidate_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047e72ff6592467eae05c0cdb0e9a570"
I20260620 21:45:58.337728 9469 raft_consensus.cc:3037] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:58.337755 9329 raft_consensus.cc:3037] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:58.338649 9469 raft_consensus.cc:2454] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449f3cea3c4f48ef9df52e1c7a5452b8 in term 1.
I20260620 21:45:58.338816 9329 raft_consensus.cc:2454] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449f3cea3c4f48ef9df52e1c7a5452b8 in term 1.
I20260620 21:45:58.338966 9540 leader_election.cc:302] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f3cea3c4f48ef9df52e1c7a5452b8, eebd1f41f3754be49a2b9b4c42bfcbc9; no voters:
I20260620 21:45:58.339138 9690 raft_consensus.cc:2781] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:58.339219 9690 raft_consensus.cc:686] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 LEADER]: Becoming Leader. State: Replica: 449f3cea3c4f48ef9df52e1c7a5452b8, State: Running, Role: LEADER
I20260620 21:45:58.339357 9690 consensus_queue.cc:237] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.339998 9713 tablet_replica.cc:439] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8: TxnStatusTablet state changed. Reason: New leader 449f3cea3c4f48ef9df52e1c7a5452b8. Latest consensus state: current_term: 1 leader_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } }
I20260620 21:45:58.340111 9713 tablet_replica.cc:442] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:45:58.340548 9725 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 21:45:58.340783 9725 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 21:45:58.341285 9170 catalog_manager.cc:5495] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 449f3cea3c4f48ef9df52e1c7a5452b8 (127.1.155.131). New cstate: current_term: 1 leader_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } health_report { overall_health: HEALTHY } } }
I20260620 21:45:58.356633 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.352353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47420 (local address 127.1.155.131:44727)
0620 21:45:58.352807 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:58.352810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:58.352823 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:45:58.352881 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:58.352883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:58.352949 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:58.353109 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:58.353394 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.354059 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.355095 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.355354 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.355470 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.355792 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:58.356206 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:58.356342 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:58.356416 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 21:45:58.356469 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":65,"threads_started":1}
I20260620 21:45:58.356617 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.352210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:44727 (local address 127.0.0.1:47420)
0620 21:45:58.352499 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:58.352522 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:58.352619 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:58.353135 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:58.353139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:58.353147 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:45:58.353275 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:58.353281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.354184 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.354188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:58.354939 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:58.354949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.355470 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.355474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:58.355573 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.355618 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:45:58.356314 (+ 696us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:45:58.356331 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 21:45:58.356396 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 21:45:58.356488 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260620 21:45:58.360108 9329 raft_consensus.cc:1261] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Refusing update from remote peer 449f3cea3c4f48ef9df52e1c7a5452b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:58.360335 9469 raft_consensus.cc:1261] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Refusing update from remote peer 449f3cea3c4f48ef9df52e1c7a5452b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:58.360687 9713 consensus_queue.cc:1035] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:58.360857 9690 consensus_queue.cc:1035] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:58.362474 9689 tablet_replica.cc:439] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570: TxnStatusTablet state changed. Reason: New leader 449f3cea3c4f48ef9df52e1c7a5452b8. Latest consensus state: current_term: 1 leader_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } }
I20260620 21:45:58.362587 9689 tablet_replica.cc:442] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:58.362613 9703 tablet_replica.cc:439] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9: TxnStatusTablet state changed. Reason: New leader 449f3cea3c4f48ef9df52e1c7a5452b8. Latest consensus state: current_term: 1 leader_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } }
I20260620 21:45:58.362730 9703 tablet_replica.cc:442] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:58.364256 9729 mvcc.cc:205] Tried to move back new op lower bound from 7299039061437919232 to 7299039061359431680. Current Snapshot: MvccSnapshot[applied={T|T < 7299039061437919232}]
I20260620 21:45:58.364429 9690 tablet_replica.cc:439] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } }
I20260620 21:45:58.364527 9690 tablet_replica.cc:442] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:45:58.364946 9703 tablet_replica.cc:439] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } }
I20260620 21:45:58.365041 9703 tablet_replica.cc:442] T d6e31a67236e45dba700f43cd58db680 P eebd1f41f3754be49a2b9b4c42bfcbc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:58.369524 9713 tablet_replica.cc:439] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } }
I20260620 21:45:58.369643 9713 tablet_replica.cc:442] T d6e31a67236e45dba700f43cd58db680 P 449f3cea3c4f48ef9df52e1c7a5452b8: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:45:58.370719 9689 tablet_replica.cc:439] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } } }
I20260620 21:45:58.370832 9689 tablet_replica.cc:442] T d6e31a67236e45dba700f43cd58db680 P 047e72ff6592467eae05c0cdb0e9a570: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:45:58.373394 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.369597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.1.155.131:44727)
0620 21:45:58.369726 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:58.369730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:58.369743 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:45:58.370053 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:58.370056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:58.370110 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:58.370159 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:58.370558 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.371272 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.372399 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.372621 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.372730 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.372958 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:58.373030 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:58.373109 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:58.373196 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 21:45:58.373253 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 21:45:58.385082 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.379943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:50283 (local address 127.1.155.131:44727)
0620 21:45:58.380057 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:58.380061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:58.382109 (+ 2048us) server_negotiation.cc:407] Connection header received
0620 21:45:58.382264 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:58.382267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:58.382306 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:58.382389 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:58.382645 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.383140 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.383999 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.384221 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.384320 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.384631 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:58.384701 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:58.384777 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:58.384861 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 21:45:58.384907 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":35}
I20260620 21:45:58.385090 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.379919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:44727 (local address 127.1.155.131:50283)
0620 21:45:58.382026 (+ 2107us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:58.382039 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:58.382166 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:58.382394 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:58.382397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:58.382404 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:45:58.382549 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:58.382554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.383270 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.383274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:58.383889 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:58.383896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.384330 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.384333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:58.384471 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.384510 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:45:58.384789 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:45:58.384796 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 21:45:58.384855 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 21:45:58.384922 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2020,"mutex_wait_us":26,"thread_start_us":94,"threads_started":1}
I20260620 21:45:58.391494 9469 raft_consensus.cc:1261] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Refusing update from remote peer 449f3cea3c4f48ef9df52e1c7a5452b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:58.391486 9329 raft_consensus.cc:1261] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Refusing update from remote peer 449f3cea3c4f48ef9df52e1c7a5452b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:58.391856 9713 consensus_queue.cc:1035] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:58.392023 9690 consensus_queue.cc:1035] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:58.434494 9170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:58.434665 9170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:58.436576 9170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb: Sending DeleteTablet for 3 replicas of tablet ce6ce915163247ae92424530fb601d92
I20260620 21:45:58.437176 9309 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6ce915163247ae92424530fb601d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:58 UTC) from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:52066
I20260620 21:45:58.437378 9449 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6ce915163247ae92424530fb601d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:58 UTC) from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:33660
I20260620 21:45:58.437414 9588 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6ce915163247ae92424530fb601d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:58 UTC) from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:47418
I20260620 21:45:58.437716 9748 tablet_replica.cc:330] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9: stopping tablet replica
I20260620 21:45:58.437888 9748 raft_consensus.cc:2229] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:58.438030 9748 raft_consensus.cc:2258] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:58.438094 9749 tablet_replica.cc:330] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8: stopping tablet replica
I20260620 21:45:58.438220 9749 raft_consensus.cc:2229] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:45:58.438500 9749 raft_consensus.cc:2258] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:58.438750 9748 ts_tablet_manager.cc:1905] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:58.439129 9749 ts_tablet_manager.cc:1905] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:58.441848 9749 ts_tablet_manager.cc:1918] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:45:58.441917 9749 log.cc:1196] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/ce6ce915163247ae92424530fb601d92
I20260620 21:45:58.442207 9747 tablet_replica.cc:330] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570: stopping tablet replica
I20260620 21:45:58.442273 9749 ts_tablet_manager.cc:1939] T ce6ce915163247ae92424530fb601d92 P 449f3cea3c4f48ef9df52e1c7a5452b8: Deleting consensus metadata
I20260620 21:45:58.442298 9748 ts_tablet_manager.cc:1918] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:45:58.442371 9748 log.cc:1196] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/ce6ce915163247ae92424530fb601d92
I20260620 21:45:58.442523 9747 raft_consensus.cc:2229] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:58.442668 9748 ts_tablet_manager.cc:1939] T ce6ce915163247ae92424530fb601d92 P eebd1f41f3754be49a2b9b4c42bfcbc9: Deleting consensus metadata
I20260620 21:45:58.442754 9747 raft_consensus.cc:2258] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:58.442950 9158 catalog_manager.cc:4841] TS 449f3cea3c4f48ef9df52e1c7a5452b8 (127.1.155.131:44727): tablet ce6ce915163247ae92424530fb601d92 (table test-table [id=e584ccfd550f4fda9d4874d43e2829e3]) successfully deleted
I20260620 21:45:58.443322 9156 catalog_manager.cc:4841] TS eebd1f41f3754be49a2b9b4c42bfcbc9 (127.1.155.130:35025): tablet ce6ce915163247ae92424530fb601d92 (table test-table [id=e584ccfd550f4fda9d4874d43e2829e3]) successfully deleted
I20260620 21:45:58.443735 9747 ts_tablet_manager.cc:1905] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:58.445981 9747 ts_tablet_manager.cc:1918] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:45:58.446072 9747 log.cc:1196] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/wal/wals/ce6ce915163247ae92424530fb601d92
I20260620 21:45:58.446313 9747 ts_tablet_manager.cc:1939] T ce6ce915163247ae92424530fb601d92 P 047e72ff6592467eae05c0cdb0e9a570: Deleting consensus metadata
I20260620 21:45:58.447093 9157 catalog_manager.cc:4841] TS 047e72ff6592467eae05c0cdb0e9a570 (127.1.155.129:36033): tablet ce6ce915163247ae92424530fb601d92 (table test-table [id=e584ccfd550f4fda9d4874d43e2829e3]) successfully deleted
Jun 20 21:45:58 dist-test-slave-5bvn krb5kdc[9127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:45:58 dist-test-slave-5bvn krb5kdc[9127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:45:58 dist-test-slave-5bvn krb5kdc[9127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 21:45:58.471189 9221 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:45:58.464096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.1.155.190:34253)
0620 21:45:58.464270 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:58.464275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:58.464488 (+ 213us) server_negotiation.cc:407] Connection header received
0620 21:45:58.464622 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:58.464624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:58.464702 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:58.464782 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:58.465669 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.466318 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.467515 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.467732 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.467809 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.471055 (+ 3246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52166: BlockingRecv error: recv got EOF from 127.0.0.1:52166 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 21:45:58 dist-test-slave-5bvn krb5kdc[9127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM
I20260620 21:45:58.481133 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.472635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.1.155.190:34253)
0620 21:45:58.472766 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:58.472773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:58.472958 (+ 185us) server_negotiation.cc:407] Connection header received
0620 21:45:58.473079 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:58.473081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:58.473129 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:58.473210 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:45:58.473762 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.474325 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.475086 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.475313 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.475422 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.477719 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:45:58.477723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:45:58.477725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:45:58.477752 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:45:58.479408 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:58.479836 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:58.479840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:58.479841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:58.479928 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:45:58.480182 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:45:58.480185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:45:58.480187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:45:58.480428 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:45:58.480565 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:58.480854 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 21:45:58.480970 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:45:58.483881 9170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:45:58.484361 9170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:58.489257 9309 tablet_service.cc:1467] Processing CreateTablet for tablet 13e660de2a1343c6b4b47fe09e924f73 (DEFAULT_TABLE table=test-table [id=569eef7931ff4437b6611079a0557611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:58.489459 9588 tablet_service.cc:1467] Processing CreateTablet for tablet 13e660de2a1343c6b4b47fe09e924f73 (DEFAULT_TABLE table=test-table [id=569eef7931ff4437b6611079a0557611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:58.489559 9309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e660de2a1343c6b4b47fe09e924f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:58.489711 9588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e660de2a1343c6b4b47fe09e924f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:58.492102 9684 tablet_bootstrap.cc:492] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570: Bootstrap starting.
I20260620 21:45:58.492108 9683 tablet_bootstrap.cc:492] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8: Bootstrap starting.
I20260620 21:45:58.492264 9449 tablet_service.cc:1467] Processing CreateTablet for tablet 13e660de2a1343c6b4b47fe09e924f73 (DEFAULT_TABLE table=test-table [id=569eef7931ff4437b6611079a0557611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:58.492517 9449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e660de2a1343c6b4b47fe09e924f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:58.492959 9684 tablet_bootstrap.cc:654] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:58.493016 9683 tablet_bootstrap.cc:654] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:58.494957 9685 tablet_bootstrap.cc:492] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9: Bootstrap starting.
I20260620 21:45:58.495734 9684 tablet_bootstrap.cc:492] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570: No bootstrap required, opened a new log
I20260620 21:45:58.495754 9685 tablet_bootstrap.cc:654] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:58.495812 9684 ts_tablet_manager.cc:1397] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260620 21:45:58.495962 9683 tablet_bootstrap.cc:492] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8: No bootstrap required, opened a new log
I20260620 21:45:58.496027 9683 ts_tablet_manager.cc:1397] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260620 21:45:58.496275 9684 raft_consensus.cc:348] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.496397 9684 raft_consensus.cc:374] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:58.496439 9684 raft_consensus.cc:729] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047e72ff6592467eae05c0cdb0e9a570, State: Initialized, Role: FOLLOWER
I20260620 21:45:58.496440 9683 raft_consensus.cc:348] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.496543 9683 raft_consensus.cc:374] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:58.496574 9683 raft_consensus.cc:729] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449f3cea3c4f48ef9df52e1c7a5452b8, State: Initialized, Role: FOLLOWER
I20260620 21:45:58.496593 9684 consensus_queue.cc:260] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.496696 9683 consensus_queue.cc:260] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.496958 9683 ts_tablet_manager.cc:1428] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:45:58.497234 9685 tablet_bootstrap.cc:492] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9: No bootstrap required, opened a new log
I20260620 21:45:58.497311 9685 ts_tablet_manager.cc:1397] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:45:58.497676 9685 raft_consensus.cc:348] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.497779 9685 raft_consensus.cc:374] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:58.498281 9684 ts_tablet_manager.cc:1428] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 21:45:58.497812 9685 raft_consensus.cc:729] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebd1f41f3754be49a2b9b4c42bfcbc9, State: Initialized, Role: FOLLOWER
I20260620 21:45:58.498651 9685 consensus_queue.cc:260] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.498932 9685 ts_tablet_manager.cc:1428] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:45:58.647398 9703 raft_consensus.cc:482] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:58.647548 9703 raft_consensus.cc:504] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.647955 9703 leader_election.cc:288] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 047e72ff6592467eae05c0cdb0e9a570 (127.1.155.129:36033), 449f3cea3c4f48ef9df52e1c7a5452b8 (127.1.155.131:44727)
I20260620 21:45:58.648386 9609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e660de2a1343c6b4b47fe09e924f73" candidate_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" is_pre_election: true
I20260620 21:45:58.648438 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e660de2a1343c6b4b47fe09e924f73" candidate_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047e72ff6592467eae05c0cdb0e9a570" is_pre_election: true
I20260620 21:45:58.648543 9609 raft_consensus.cc:2454] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebd1f41f3754be49a2b9b4c42bfcbc9 in term 0.
I20260620 21:45:58.648573 9329 raft_consensus.cc:2454] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebd1f41f3754be49a2b9b4c42bfcbc9 in term 0.
I20260620 21:45:58.648835 9402 leader_election.cc:302] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f3cea3c4f48ef9df52e1c7a5452b8, eebd1f41f3754be49a2b9b4c42bfcbc9; no voters:
I20260620 21:45:58.649011 9703 raft_consensus.cc:2781] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:58.649094 9703 raft_consensus.cc:482] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:58.649149 9703 raft_consensus.cc:3037] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:58.650224 9703 raft_consensus.cc:504] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.650579 9703 leader_election.cc:288] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [CANDIDATE]: Term 1 election: Requested vote from peers 047e72ff6592467eae05c0cdb0e9a570 (127.1.155.129:36033), 449f3cea3c4f48ef9df52e1c7a5452b8 (127.1.155.131:44727)
I20260620 21:45:58.650877 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e660de2a1343c6b4b47fe09e924f73" candidate_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047e72ff6592467eae05c0cdb0e9a570"
I20260620 21:45:58.651010 9329 raft_consensus.cc:3037] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:58.651851 9329 raft_consensus.cc:2454] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebd1f41f3754be49a2b9b4c42bfcbc9 in term 1.
I20260620 21:45:58.652107 9609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e660de2a1343c6b4b47fe09e924f73" candidate_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8"
I20260620 21:45:58.652133 9401 leader_election.cc:302] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047e72ff6592467eae05c0cdb0e9a570, eebd1f41f3754be49a2b9b4c42bfcbc9; no voters:
I20260620 21:45:58.652251 9609 raft_consensus.cc:3037] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:58.653113 9609 raft_consensus.cc:2454] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebd1f41f3754be49a2b9b4c42bfcbc9 in term 1.
I20260620 21:45:58.653323 9703 raft_consensus.cc:2781] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:58.653539 9703 raft_consensus.cc:686] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 LEADER]: Becoming Leader. State: Replica: eebd1f41f3754be49a2b9b4c42bfcbc9, State: Running, Role: LEADER
I20260620 21:45:58.653699 9703 consensus_queue.cc:237] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.655323 9171 catalog_manager.cc:5495] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to eebd1f41f3754be49a2b9b4c42bfcbc9 (127.1.155.130). New cstate: current_term: 1 leader_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } health_report { overall_health: UNKNOWN } } }
I20260620 21:45:58.689432 9768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.686555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.1.155.130:35025)
0620 21:45:58.686821 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:58.686828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:58.686840 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:45:58.686882 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:58.686884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:58.686924 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:58.686995 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:58.687217 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.687730 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.688496 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.688712 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.688783 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.688989 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:58.689050 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:58.689100 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:58.689182 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 21:45:58.689224 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":116,"threads_started":1}
I20260620 21:45:58.691437 9609 raft_consensus.cc:1261] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Refusing update from remote peer eebd1f41f3754be49a2b9b4c42bfcbc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:58.691437 9329 raft_consensus.cc:1261] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Refusing update from remote peer eebd1f41f3754be49a2b9b4c42bfcbc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:58.691866 9703 consensus_queue.cc:1035] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:58.692050 9765 consensus_queue.cc:1035] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:58.698536 9172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:58.698695 9172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:58.700814 9172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb: Sending DeleteTablet for 3 replicas of tablet 13e660de2a1343c6b4b47fe09e924f73
I20260620 21:45:58.701373 9449 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e660de2a1343c6b4b47fe09e924f73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:58 UTC) from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:33660
I20260620 21:45:58.701448 9309 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e660de2a1343c6b4b47fe09e924f73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:58 UTC) from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:52066
I20260620 21:45:58.701503 9588 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e660de2a1343c6b4b47fe09e924f73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:58 UTC) from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:47418
I20260620 21:45:58.701543 9748 tablet_replica.cc:330] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9: stopping tablet replica
I20260620 21:45:58.701588 9747 tablet_replica.cc:330] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570: stopping tablet replica
I20260620 21:45:58.701675 9747 raft_consensus.cc:2229] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:58.701648 9748 raft_consensus.cc:2229] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:45:58.701781 9747 raft_consensus.cc:2258] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:58.701869 9748 raft_consensus.cc:2258] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:58.701893 9749 tablet_replica.cc:330] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8: stopping tablet replica
I20260620 21:45:58.702008 9749 raft_consensus.cc:2229] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:58.702138 9749 raft_consensus.cc:2258] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:58.702520 9747 ts_tablet_manager.cc:1905] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:58.702474 9172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 21:45:58.702739 9749 ts_tablet_manager.cc:1905] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 21:45:58.702852 9172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:45:58.704566 9747 ts_tablet_manager.cc:1918] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:45:58.704632 9747 log.cc:1196] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/wal/wals/13e660de2a1343c6b4b47fe09e924f73
I20260620 21:45:58.704908 9747 ts_tablet_manager.cc:1939] T 13e660de2a1343c6b4b47fe09e924f73 P 047e72ff6592467eae05c0cdb0e9a570: Deleting consensus metadata
I20260620 21:45:58.705092 9749 ts_tablet_manager.cc:1918] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:45:58.705168 9749 log.cc:1196] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/13e660de2a1343c6b4b47fe09e924f73
I20260620 21:45:58.705457 9749 ts_tablet_manager.cc:1939] T 13e660de2a1343c6b4b47fe09e924f73 P 449f3cea3c4f48ef9df52e1c7a5452b8: Deleting consensus metadata
I20260620 21:45:58.706100 9748 ts_tablet_manager.cc:1905] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:58.706382 9158 catalog_manager.cc:4841] TS 449f3cea3c4f48ef9df52e1c7a5452b8 (127.1.155.131:44727): tablet 13e660de2a1343c6b4b47fe09e924f73 (table test-table [id=569eef7931ff4437b6611079a0557611]) successfully deleted
I20260620 21:45:58.707266 9449 tablet_service.cc:1467] Processing CreateTablet for tablet cb5a583735404e7f89bc8d16aa08ad93 (DEFAULT_TABLE table=test-table [id=35f134d213c3453c915cbc2c46532e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:58.707352 9588 tablet_service.cc:1467] Processing CreateTablet for tablet cb5a583735404e7f89bc8d16aa08ad93 (DEFAULT_TABLE table=test-table [id=35f134d213c3453c915cbc2c46532e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:58.707535 9449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb5a583735404e7f89bc8d16aa08ad93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:58.707614 9588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb5a583735404e7f89bc8d16aa08ad93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:58.707800 9157 catalog_manager.cc:4841] TS 047e72ff6592467eae05c0cdb0e9a570 (127.1.155.129:36033): tablet 13e660de2a1343c6b4b47fe09e924f73 (table test-table [id=569eef7931ff4437b6611079a0557611]) successfully deleted
I20260620 21:45:58.708249 9309 tablet_service.cc:1467] Processing CreateTablet for tablet cb5a583735404e7f89bc8d16aa08ad93 (DEFAULT_TABLE table=test-table [id=35f134d213c3453c915cbc2c46532e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:45:58.708462 9309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb5a583735404e7f89bc8d16aa08ad93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:45:58.708825 9748 ts_tablet_manager.cc:1918] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:45:58.708901 9748 log.cc:1196] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/13e660de2a1343c6b4b47fe09e924f73
I20260620 21:45:58.709133 9748 ts_tablet_manager.cc:1939] T 13e660de2a1343c6b4b47fe09e924f73 P eebd1f41f3754be49a2b9b4c42bfcbc9: Deleting consensus metadata
I20260620 21:45:58.709741 9685 tablet_bootstrap.cc:492] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9: Bootstrap starting.
I20260620 21:45:58.709822 9683 tablet_bootstrap.cc:492] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8: Bootstrap starting.
I20260620 21:45:58.710517 9683 tablet_bootstrap.cc:654] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:58.711241 9684 tablet_bootstrap.cc:492] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570: Bootstrap starting.
I20260620 21:45:58.712114 9684 tablet_bootstrap.cc:654] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:58.713201 9684 tablet_bootstrap.cc:492] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570: No bootstrap required, opened a new log
I20260620 21:45:58.713287 9684 ts_tablet_manager.cc:1397] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 21:45:58.713378 9685 tablet_bootstrap.cc:654] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9: Neither blocks nor log segments found. Creating new log.
I20260620 21:45:58.713744 9684 raft_consensus.cc:348] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.713896 9684 raft_consensus.cc:374] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:58.713939 9684 raft_consensus.cc:729] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047e72ff6592467eae05c0cdb0e9a570, State: Initialized, Role: FOLLOWER
I20260620 21:45:58.713927 9156 catalog_manager.cc:4841] TS eebd1f41f3754be49a2b9b4c42bfcbc9 (127.1.155.130:35025): tablet 13e660de2a1343c6b4b47fe09e924f73 (table test-table [id=569eef7931ff4437b6611079a0557611]) successfully deleted
I20260620 21:45:58.714067 9684 consensus_queue.cc:260] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.714341 9684 ts_tablet_manager.cc:1428] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:58.714457 9685 tablet_bootstrap.cc:492] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9: No bootstrap required, opened a new log
I20260620 21:45:58.714533 9685 ts_tablet_manager.cc:1397] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 21:45:58.714905 9685 raft_consensus.cc:348] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.715026 9685 raft_consensus.cc:374] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:58.715087 9685 raft_consensus.cc:729] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebd1f41f3754be49a2b9b4c42bfcbc9, State: Initialized, Role: FOLLOWER
I20260620 21:45:58.715212 9685 consensus_queue.cc:260] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.715461 9685 ts_tablet_manager.cc:1428] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:45:58.716451 9683 tablet_bootstrap.cc:492] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8: No bootstrap required, opened a new log
I20260620 21:45:58.716522 9683 ts_tablet_manager.cc:1397] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260620 21:45:58.716909 9683 raft_consensus.cc:348] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.717025 9683 raft_consensus.cc:374] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:45:58.717067 9683 raft_consensus.cc:729] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449f3cea3c4f48ef9df52e1c7a5452b8, State: Initialized, Role: FOLLOWER
I20260620 21:45:58.717195 9683 consensus_queue.cc:260] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.717444 9683 ts_tablet_manager.cc:1428] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:45:58.863902 9689 raft_consensus.cc:482] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:45:58.864096 9689 raft_consensus.cc:504] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.865221 9689 leader_election.cc:288] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebd1f41f3754be49a2b9b4c42bfcbc9 (127.1.155.130:35025), 449f3cea3c4f48ef9df52e1c7a5452b8 (127.1.155.131:44727)
I20260620 21:45:58.869107 9768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.865486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:47581 (local address 127.1.155.130:35025)
0620 21:45:58.865691 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:58.865696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:58.865849 (+ 153us) server_negotiation.cc:407] Connection header received
0620 21:45:58.866012 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:58.866015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:58.866062 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:58.866144 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:58.866446 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.866951 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.868132 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.868537 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.868674 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.868725 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:58.868912 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 21:45:58.868981 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 21:45:58.869123 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.865563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:47201 (local address 127.1.155.131:44727)
0620 21:45:58.865699 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:58.865702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:58.865896 (+ 194us) server_negotiation.cc:407] Connection header received
0620 21:45:58.866053 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:58.866057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:58.866108 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:58.866193 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:45:58.866475 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.866958 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.868107 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.868517 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.868652 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.868709 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:58.868934 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 21:45:58.869007 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 21:45:58.869144 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.865385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:35025 (local address 127.1.155.129:47581)
0620 21:45:58.865731 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:58.865745 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:58.865892 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:58.866156 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:58.866158 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:58.866174 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:58.866336 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:58.866342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.867090 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.867094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:58.867995 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:58.868016 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.868680 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.868684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:58.868779 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.868800 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:45:58.868897 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 21:45:58.868970 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":51,"spinlock_wait_cycles":11520,"thread_start_us":71,"threads_started":1}
I20260620 21:45:58.869156 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.865475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:44727 (local address 127.1.155.129:47201)
0620 21:45:58.865788 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:58.865799 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:58.865939 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:58.866204 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:58.866207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:58.866236 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:45:58.866370 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:58.866375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.867068 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.867072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:58.867981 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:58.867990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.868641 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.868645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:58.868748 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.868820 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 21:45:58.868905 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 21:45:58.868979 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":22,"thread_start_us":65,"threads_started":1}
I20260620 21:45:58.869872 9609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb5a583735404e7f89bc8d16aa08ad93" candidate_uuid: "047e72ff6592467eae05c0cdb0e9a570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" is_pre_election: true
I20260620 21:45:58.869947 9469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb5a583735404e7f89bc8d16aa08ad93" candidate_uuid: "047e72ff6592467eae05c0cdb0e9a570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" is_pre_election: true
I20260620 21:45:58.870040 9609 raft_consensus.cc:2454] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 047e72ff6592467eae05c0cdb0e9a570 in term 0.
I20260620 21:45:58.870111 9469 raft_consensus.cc:2454] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 047e72ff6592467eae05c0cdb0e9a570 in term 0.
I20260620 21:45:58.870416 9262 leader_election.cc:302] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047e72ff6592467eae05c0cdb0e9a570, 449f3cea3c4f48ef9df52e1c7a5452b8; no voters:
I20260620 21:45:58.870700 9689 raft_consensus.cc:2781] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:45:58.870765 9689 raft_consensus.cc:482] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:45:58.870803 9689 raft_consensus.cc:3037] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:58.871789 9689 raft_consensus.cc:504] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.872212 9689 leader_election.cc:288] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [CANDIDATE]: Term 1 election: Requested vote from peers eebd1f41f3754be49a2b9b4c42bfcbc9 (127.1.155.130:35025), 449f3cea3c4f48ef9df52e1c7a5452b8 (127.1.155.131:44727)
I20260620 21:45:58.872488 9469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb5a583735404e7f89bc8d16aa08ad93" candidate_uuid: "047e72ff6592467eae05c0cdb0e9a570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9"
I20260620 21:45:58.872560 9609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb5a583735404e7f89bc8d16aa08ad93" candidate_uuid: "047e72ff6592467eae05c0cdb0e9a570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8"
I20260620 21:45:58.872602 9469 raft_consensus.cc:3037] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:58.872699 9609 raft_consensus.cc:3037] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:45:58.873533 9469 raft_consensus.cc:2454] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 047e72ff6592467eae05c0cdb0e9a570 in term 1.
I20260620 21:45:58.873625 9609 raft_consensus.cc:2454] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 047e72ff6592467eae05c0cdb0e9a570 in term 1.
I20260620 21:45:58.873876 9260 leader_election.cc:302] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047e72ff6592467eae05c0cdb0e9a570, eebd1f41f3754be49a2b9b4c42bfcbc9; no voters:
I20260620 21:45:58.874040 9689 raft_consensus.cc:2781] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:45:58.874279 9689 raft_consensus.cc:686] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 LEADER]: Becoming Leader. State: Replica: 047e72ff6592467eae05c0cdb0e9a570, State: Running, Role: LEADER
I20260620 21:45:58.874526 9689 consensus_queue.cc:237] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } }
I20260620 21:45:58.876793 9172 catalog_manager.cc:5495] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 reported cstate change: term changed from 0 to 1, leader changed from <none> to 047e72ff6592467eae05c0cdb0e9a570 (127.1.155.129). New cstate: current_term: 1 leader_uuid: "047e72ff6592467eae05c0cdb0e9a570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047e72ff6592467eae05c0cdb0e9a570" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 } health_report { overall_health: UNKNOWN } } }
I20260620 21:45:58.912972 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.909325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.1.155.129:36033)
0620 21:45:58.909624 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:58.909633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:58.909643 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:45:58.909708 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:58.909710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:58.909759 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:58.909933 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:58.910171 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.910704 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.911518 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.911758 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.911864 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.912117 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:58.912564 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:58.912691 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:58.912762 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 21:45:58.912812 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":126,"threads_started":1}
I20260620 21:45:58.922597 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.919551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:54469 (local address 127.1.155.131:44727)
0620 21:45:58.919668 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:58.919673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:58.919830 (+ 157us) server_negotiation.cc:407] Connection header received
0620 21:45:58.919915 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:58.919938 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:58.919995 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:58.920061 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:58.920273 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.920790 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.921627 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.921876 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.921963 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.922201 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:58.922259 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:58.922313 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:58.922454 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 21:45:58.922496 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:45:58.922657 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.919445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:44727 (local address 127.1.155.129:54469)
0620 21:45:58.919747 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:58.919760 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:58.919844 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:58.920069 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:58.920071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:58.920076 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:45:58.920180 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:58.920183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.920879 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.920881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:58.921517 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:58.921527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.921980 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.921983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:58.922052 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.922094 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:45:58.922365 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:45:58.922371 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:45:58.922438 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 21:45:58.922497 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":127,"threads_started":1}
I20260620 21:45:58.928683 9609 raft_consensus.cc:1261] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Refusing update from remote peer 047e72ff6592467eae05c0cdb0e9a570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:58.928714 9469 raft_consensus.cc:1261] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Refusing update from remote peer 047e72ff6592467eae05c0cdb0e9a570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:45:58.929149 9689 consensus_queue.cc:1035] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebd1f41f3754be49a2b9b4c42bfcbc9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:58.929333 9777 consensus_queue.cc:1035] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449f3cea3c4f48ef9df52e1c7a5452b8" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:45:58.946391 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.943065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:51521 (local address 127.1.155.129:36033)
0620 21:45:58.943180 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 21:45:58.943184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:45:58.943390 (+ 206us) server_negotiation.cc:407] Connection header received
0620 21:45:58.943514 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:45:58.943517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:45:58.943600 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:45:58.943690 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:45:58.943951 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.944446 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.945235 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.945562 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.945678 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.945944 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:45:58.946023 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:45:58.946107 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 21:45:58.946207 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 21:45:58.946257 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:45:58.946434 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:45:58.943018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:36033 (local address 127.1.155.131:51521)
0620 21:45:58.943311 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 21:45:58.943323 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:45:58.943402 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:45:58.943703 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:45:58.943705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:45:58.943717 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:45:58.943828 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:58.943835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.944531 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.944533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:58.945127 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:45:58.945134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:45:58.945681 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:45:58.945684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:45:58.945766 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:45:58.945805 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:45:58.946130 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:45:58.946137 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 21:45:58.946195 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 21:45:58.946257 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":84,"threads_started":1}
I20260620 21:45:58.968403 9171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:58.968575 9171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:45:58.970853 9171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 389ae4b24e25449d8ea659d032e2d5eb: Sending DeleteTablet for 3 replicas of tablet cb5a583735404e7f89bc8d16aa08ad93
I20260620 21:45:58.971558 9309 tablet_service.cc:1514] Processing DeleteTablet for tablet cb5a583735404e7f89bc8d16aa08ad93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:58 UTC) from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:52066
I20260620 21:45:58.971558 9449 tablet_service.cc:1514] Processing DeleteTablet for tablet cb5a583735404e7f89bc8d16aa08ad93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:58 UTC) from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:33660
I20260620 21:45:58.971763 9748 tablet_replica.cc:330] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9: stopping tablet replica
I20260620 21:45:58.971763 9747 tablet_replica.cc:330] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570: stopping tablet replica
I20260620 21:45:58.971781 9588 tablet_service.cc:1514] Processing DeleteTablet for tablet cb5a583735404e7f89bc8d16aa08ad93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:45:58 UTC) from {username='oryx', principal='oryx/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:47418
I20260620 21:45:58.971899 9747 raft_consensus.cc:2229] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:45:58.971908 9748 raft_consensus.cc:2229] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:58.971956 9749 tablet_replica.cc:330] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8: stopping tablet replica
I20260620 21:45:58.972023 9748 raft_consensus.cc:2258] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:58.972079 9749 raft_consensus.cc:2229] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:45:58.972194 9749 raft_consensus.cc:2258] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:58.972208 9747 raft_consensus.cc:2258] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:45:58.972982 9748 ts_tablet_manager.cc:1905] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:58.973043 9749 ts_tablet_manager.cc:1905] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:58.973237 9747 ts_tablet_manager.cc:1905] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:45:58.975596 9749 ts_tablet_manager.cc:1918] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:45:58.975590 9747 ts_tablet_manager.cc:1918] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:45:58.975620 9748 ts_tablet_manager.cc:1918] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:45:58.975663 9749 log.cc:1196] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/cb5a583735404e7f89bc8d16aa08ad93
I20260620 21:45:58.975670 9747 log.cc:1196] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-0/wal/wals/cb5a583735404e7f89bc8d16aa08ad93
I20260620 21:45:58.975700 9748 log.cc:1196] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/cb5a583735404e7f89bc8d16aa08ad93
I20260620 21:45:58.975936 9747 ts_tablet_manager.cc:1939] T cb5a583735404e7f89bc8d16aa08ad93 P 047e72ff6592467eae05c0cdb0e9a570: Deleting consensus metadata
I20260620 21:45:58.975936 9749 ts_tablet_manager.cc:1939] T cb5a583735404e7f89bc8d16aa08ad93 P 449f3cea3c4f48ef9df52e1c7a5452b8: Deleting consensus metadata
I20260620 21:45:58.975968 9748 ts_tablet_manager.cc:1939] T cb5a583735404e7f89bc8d16aa08ad93 P eebd1f41f3754be49a2b9b4c42bfcbc9: Deleting consensus metadata
I20260620 21:45:58.976933 9157 catalog_manager.cc:4841] TS 047e72ff6592467eae05c0cdb0e9a570 (127.1.155.129:36033): tablet cb5a583735404e7f89bc8d16aa08ad93 (table test-table [id=35f134d213c3453c915cbc2c46532e77]) successfully deleted
I20260620 21:45:58.977252 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 9244
I20260620 21:45:58.978030 9158 catalog_manager.cc:4841] TS 449f3cea3c4f48ef9df52e1c7a5452b8 (127.1.155.131:44727): tablet cb5a583735404e7f89bc8d16aa08ad93 (table test-table [id=35f134d213c3453c915cbc2c46532e77]) successfully deleted
I20260620 21:45:58.978958 9156 catalog_manager.cc:4841] TS eebd1f41f3754be49a2b9b4c42bfcbc9 (127.1.155.130:35025): tablet cb5a583735404e7f89bc8d16aa08ad93 (table test-table [id=35f134d213c3453c915cbc2c46532e77]) successfully deleted
I20260620 21:45:58.984318 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 9384
I20260620 21:45:58.990896 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 9524
I20260620 21:45:58.997370 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 9140
2026-06-20T21:45:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3460 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:45:59 dist-test-slave-5bvn krb5kdc[9792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:45:59 dist-test-slave-5bvn krb5kdc[9792](info): set up 2 sockets
Jun 20 21:45:59 dist-test-slave-5bvn krb5kdc[9792](info): commencing operation
krb5kdc: starting...
W20260620 21:46:00.046886 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:46:00 dist-test-slave-5bvn krb5kdc[9792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:46:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:46:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:46:00.180806 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:41327
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:41327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:46:00.295439 9805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:00.295944 9805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:00.296130 9805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:00.300326 9805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:46:00.300424 9805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:00.300463 9805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:00.300531 9805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:46:00.300587 9805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:46:00.306253 9805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:41327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:41327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:00.307333 9805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:00.308355 9805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:00.316542 9813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:00.316548 9810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:00.316597 9811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:00.316917 9805 server_base.cc:1034] running on GCE node
I20260620 21:46:00.317519 9805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:00.318631 9805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:00.319815 9805 hybrid_clock.cc:648] HybridClock initialized: now 1781991960319785 us; error 45 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T21:46:00Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1330 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:46:00 dist-test-slave-5bvn krb5kdc[9818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:46:00 dist-test-slave-5bvn krb5kdc[9818](info): set up 2 sockets
Jun 20 21:46:00 dist-test-slave-5bvn krb5kdc[9818](info): commencing operation
krb5kdc: starting...
W20260620 21:46:01.378602 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:46:01 dist-test-slave-5bvn krb5kdc[9818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:46:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:46:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:46:01.500782 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:46155
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:46155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:46:01.613389 9831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:01.613660 9831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:01.613700 9831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:01.617908 9831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:46:01.617969 9831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:01.617988 9831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:01.618005 9831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:46:01.618054 9831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:46:01.623600 9831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:46155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:46155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:01.624683 9831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:01.625705 9831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:01.635597 9837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:01.635644 9836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:01.635712 9831 server_base.cc:1034] running on GCE node
W20260620 21:46:01.635582 9839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:01.636452 9831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:01.637822 9831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:01.639092 9831 hybrid_clock.cc:648] HybridClock initialized: now 1781991961639062 us; error 42 us; skew 500 ppm
Jun 20 21:46:01 dist-test-slave-5bvn krb5kdc[9818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991961, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:46:01.646773 9831 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:46:01.647825 9831 webserver.cc:460] Webserver started at http://127.1.155.190:34859/ using document root <none> and password file <none>
I20260620 21:46:01.648427 9831 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:01.648520 9831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:01.648763 9831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:01.651005 9831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "275e5a201e1d4cf28791dc1537bc48c3"
format_stamp: "Formatted at 2026-06-20 21:46:01 on dist-test-slave-5bvn"
server_key: "6f71877de08c152fbb0c7d58f2030e64"
server_key_iv: "05f033f10ca03fe3e642fae527a1562b"
server_key_version: "encryptionkey@0"
I20260620 21:46:01.651641 9831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "275e5a201e1d4cf28791dc1537bc48c3"
format_stamp: "Formatted at 2026-06-20 21:46:01 on dist-test-slave-5bvn"
server_key: "6f71877de08c152fbb0c7d58f2030e64"
server_key_iv: "05f033f10ca03fe3e642fae527a1562b"
server_key_version: "encryptionkey@0"
I20260620 21:46:01.655659 9831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 21:46:01.658304 9846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:01.659622 9831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:46:01.659789 9831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "275e5a201e1d4cf28791dc1537bc48c3"
format_stamp: "Formatted at 2026-06-20 21:46:01 on dist-test-slave-5bvn"
server_key: "6f71877de08c152fbb0c7d58f2030e64"
server_key_iv: "05f033f10ca03fe3e642fae527a1562b"
server_key_version: "encryptionkey@0"
I20260620 21:46:01.659916 9831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:01.677583 9831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:01.681727 9831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:01.681995 9831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:01.689512 9831 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:46155
I20260620 21:46:01.689553 9898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:46155 every 8 connection(s)
I20260620 21:46:01.690575 9831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:46:01.693486 9899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:46:01.697214 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 9831
I20260620 21:46:01.697315 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:46:01.697655 1646 external_mini_cluster.cc:1428] Setting key 455bad57caa63f0591265772d829244e
I20260620 21:46:01.699800 9899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3: Bootstrap starting.
I20260620 21:46:01.702487 9899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3: Neither blocks nor log segments found. Creating new log.
I20260620 21:46:01.703331 9899 log.cc:824] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3: Log is configured to *not* fsync() on all Append() calls
I20260620 21:46:01.705607 9899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3: No bootstrap required, opened a new log
Jun 20 21:46:01 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:46:01.709517 9899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275e5a201e1d4cf28791dc1537bc48c3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46155 } }
I20260620 21:46:01.709789 9899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:46:01.709900 9899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275e5a201e1d4cf28791dc1537bc48c3, State: Initialized, Role: FOLLOWER
I20260620 21:46:01.710407 9899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275e5a201e1d4cf28791dc1537bc48c3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46155 } }
I20260620 21:46:01.710578 9899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:46:01.710667 9899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:46:01.710788 9899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:46:01.711854 9899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275e5a201e1d4cf28791dc1537bc48c3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46155 } }
I20260620 21:46:01.712242 9899 leader_election.cc:302] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 275e5a201e1d4cf28791dc1537bc48c3; no voters:
I20260620 21:46:01.712581 9899 leader_election.cc:288] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:46:01.712702 9904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:46:01.712920 9904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 1 LEADER]: Becoming Leader. State: Replica: 275e5a201e1d4cf28791dc1537bc48c3, State: Running, Role: LEADER
I20260620 21:46:01.713258 9904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275e5a201e1d4cf28791dc1537bc48c3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46155 } }
I20260620 21:46:01.713673 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:01.699206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35296 (local address 127.1.155.190:46155)
0620 21:46:01.699648 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:01.699664 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:01.699698 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:46:01.700382 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:01.700402 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:01.700749 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:01.701030 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:01.702026 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:01.703105 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:01.704239 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:01.704707 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:01.704825 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:01.707744 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:01.707749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:01.707764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:46:01.707789 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:01.710126 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:01.710682 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:01.710689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:01.710699 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:01.710845 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:01.711181 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:01.711183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:01.711184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:01.711572 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:01.711727 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:01.712236 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 21:46:01.712587 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":160,"threads_started":1}
I20260620 21:46:01.713954 9899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:46:01.715197 9905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "275e5a201e1d4cf28791dc1537bc48c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275e5a201e1d4cf28791dc1537bc48c3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46155 } } }
I20260620 21:46:01.715332 9905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [sys.catalog]: This master's current role is: LEADER
I20260620 21:46:01.715574 9905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 275e5a201e1d4cf28791dc1537bc48c3. Latest consensus state: current_term: 1 leader_uuid: "275e5a201e1d4cf28791dc1537bc48c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275e5a201e1d4cf28791dc1537bc48c3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46155 } } }
I20260620 21:46:01.715658 9905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [sys.catalog]: This master's current role is: LEADER
I20260620 21:46:01.716181 9913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:46:01.720466 9913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:46:01.728716 9913 catalog_manager.cc:1347] Generated new cluster ID: 6566ac45940543b194fdd01981bd13ea
I20260620 21:46:01.728822 9913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:46:01.750342 9913 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:46:01.751848 9913 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:46:01.758908 9913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3: Generated new TSK 0
I20260620 21:46:01.759858 9913 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:46:01.795140 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46155
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:46:01.930914 9925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:01.931174 9925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:01.931218 9925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:01.936168 9925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:01.936255 9925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:01.936338 9925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:46:01.943243 9925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:01.944638 9925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:01.946009 9925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:01.956650 9931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:01.956688 9925 server_base.cc:1034] running on GCE node
W20260620 21:46:01.956650 9933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:01.956650 9930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:01.957293 9925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:01.957957 9925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:01.959111 9925 hybrid_clock.cc:648] HybridClock initialized: now 1781991961959085 us; error 41 us; skew 500 ppm
Jun 20 21:46:01 dist-test-slave-5bvn krb5kdc[9818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991961, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:46:01.961766 9925 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:46:01.962666 9925 webserver.cc:460] Webserver started at http://127.1.155.129:35347/ using document root <none> and password file <none>
I20260620 21:46:01.963245 9925 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:01.963308 9925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:01.963524 9925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:01.965804 9925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "ad13b76e05f84e16bf71e3c602a2e510"
format_stamp: "Formatted at 2026-06-20 21:46:01 on dist-test-slave-5bvn"
server_key: "6bf3e8910b80a900b3082c2fe02b8321"
server_key_iv: "f30663721aee596e6b3f9cd24f238a1c"
server_key_version: "encryptionkey@0"
I20260620 21:46:01.966431 9925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "ad13b76e05f84e16bf71e3c602a2e510"
format_stamp: "Formatted at 2026-06-20 21:46:01 on dist-test-slave-5bvn"
server_key: "6bf3e8910b80a900b3082c2fe02b8321"
server_key_iv: "f30663721aee596e6b3f9cd24f238a1c"
server_key_version: "encryptionkey@0"
I20260620 21:46:01.970197 9925 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:46:01.972669 9940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:01.974107 9925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:46:01.974208 9925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "ad13b76e05f84e16bf71e3c602a2e510"
format_stamp: "Formatted at 2026-06-20 21:46:01 on dist-test-slave-5bvn"
server_key: "6bf3e8910b80a900b3082c2fe02b8321"
server_key_iv: "f30663721aee596e6b3f9cd24f238a1c"
server_key_version: "encryptionkey@0"
I20260620 21:46:01.974350 9925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:01.985158 9925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:01.988057 9925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:01.988260 9925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:01.988843 9925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:46:01.989647 9925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:46:01.989720 9925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:01.989791 9925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:46:01.989835 9925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:01.998358 9925 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:40857
I20260620 21:46:01.998412 10053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:40857 every 8 connection(s)
I20260620 21:46:01.999270 9925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:46:02.002667 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 9925
I20260620 21:46:02.002821 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:46:02.003098 1646 external_mini_cluster.cc:1428] Setting key 41d9c2bb21aa832a99220605ca01a90b
Jun 20 21:46:02 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991961, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
I20260620 21:46:02.014710 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:02.001154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:34007 (local address 127.1.155.190:46155)
0620 21:46:02.001283 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:02.001286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:02.002343 (+ 1057us) server_negotiation.cc:407] Connection header received
0620 21:46:02.003331 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:02.003335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:02.003392 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:02.003475 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:02.004890 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.005402 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.006738 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.007155 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.007283 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:02.010251 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:02.010255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:02.010260 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:46:02.010291 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:02.012160 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:02.012841 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:02.012847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:02.012850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:02.012945 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:02.013354 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:02.013357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:02.013358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:02.013665 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:02.013743 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:02.014367 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 21:46:02.014578 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
I20260620 21:46:02.015637 10056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:02.001489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:34007)
0620 21:46:02.002195 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:02.002233 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:02.003102 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:02.003667 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:02.003676 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:02.004053 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:02.004738 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:02.004760 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.005528 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.005534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:02.006612 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:02.006621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.007358 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.007362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:02.007508 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:02.008127 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:02.008143 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:02.010071 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:46:02.012311 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:46:02.012317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:46:02.012331 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:46:02.012623 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:46:02.013045 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:46:02.013048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:46:02.013049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:46:02.013208 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:46:02.013749 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:46:02.013754 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:46:02.014093 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 21:46:02.014253 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 21:46:02.014613 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":113,"threads_started":1}
I20260620 21:46:02.018185 10054 heartbeater.cc:347] Connected to a master server at 127.1.155.190:46155
I20260620 21:46:02.018455 10054 heartbeater.cc:464] Registering TS with master...
I20260620 21:46:02.018960 10054 heartbeater.cc:511] Master 127.1.155.190:46155 requested a full tablet report, sending...
I20260620 21:46:02.020478 9863 ts_manager.cc:194] Registered new tserver with Master: ad13b76e05f84e16bf71e3c602a2e510 (127.1.155.129:40857)
I20260620 21:46:02.021596 9863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:34007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:46:02.032539 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46155
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:46:02.145516 10059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:02.146134 10059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:02.146217 10059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:02.150298 10059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:02.150348 10059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:02.150439 10059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:46:02.155974 10059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:02.157014 10059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:02.158113 10059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:02.168145 10067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:02.168154 10064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:02.168125 10065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:02.168413 10059 server_base.cc:1034] running on GCE node
I20260620 21:46:02.168956 10059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:02.169612 10059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:02.170832 10059 hybrid_clock.cc:648] HybridClock initialized: now 1781991962170813 us; error 31 us; skew 500 ppm
Jun 20 21:46:02 dist-test-slave-5bvn krb5kdc[9818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:46:02.173600 10059 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:46:02.174533 10059 webserver.cc:460] Webserver started at http://127.1.155.130:45935/ using document root <none> and password file <none>
I20260620 21:46:02.175094 10059 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:02.175184 10059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:02.175406 10059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:02.177423 10059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "753490598e3349f58d1e5b2cdd43eb84"
format_stamp: "Formatted at 2026-06-20 21:46:02 on dist-test-slave-5bvn"
server_key: "2be90ef94b5eff4096a817d7d73ff638"
server_key_iv: "173a1a50c0d8ca6045dd95e591f33e0a"
server_key_version: "encryptionkey@0"
I20260620 21:46:02.178107 10059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "753490598e3349f58d1e5b2cdd43eb84"
format_stamp: "Formatted at 2026-06-20 21:46:02 on dist-test-slave-5bvn"
server_key: "2be90ef94b5eff4096a817d7d73ff638"
server_key_iv: "173a1a50c0d8ca6045dd95e591f33e0a"
server_key_version: "encryptionkey@0"
I20260620 21:46:02.181948 10059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260620 21:46:02.184449 10074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:02.185681 10059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:46:02.185789 10059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "753490598e3349f58d1e5b2cdd43eb84"
format_stamp: "Formatted at 2026-06-20 21:46:02 on dist-test-slave-5bvn"
server_key: "2be90ef94b5eff4096a817d7d73ff638"
server_key_iv: "173a1a50c0d8ca6045dd95e591f33e0a"
server_key_version: "encryptionkey@0"
I20260620 21:46:02.185979 10059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:02.208611 10059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:02.211865 10059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:02.212080 10059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:02.212692 10059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:46:02.213544 10059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:46:02.213593 10059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:02.213660 10059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:46:02.213701 10059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:02.223570 10059 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:33329
I20260620 21:46:02.223589 10187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:33329 every 8 connection(s)
I20260620 21:46:02.224484 10059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:46:02.229008 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 10059
I20260620 21:46:02.229139 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:46:02.229398 1646 external_mini_cluster.cc:1428] Setting key 01c324d36174d56abc823dfdfd15dc12
Jun 20 21:46:02 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:46:02.238574 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:02.226241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:33439 (local address 127.1.155.190:46155)
0620 21:46:02.226393 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:02.226397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:02.227413 (+ 1016us) server_negotiation.cc:407] Connection header received
0620 21:46:02.228302 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:02.228305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:02.228373 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:02.228474 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:02.229715 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.230386 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.231604 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.231833 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.231945 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:02.234748 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:02.234752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:02.234754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:46:02.234783 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:02.236252 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:02.236871 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:02.236874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:02.236875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:02.236949 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:02.237361 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:02.237363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:02.237364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:02.237535 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:02.237610 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:02.238341 (+ 731us) server_negotiation.cc:299] Negotiation successful
0620 21:46:02.238449 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
I20260620 21:46:02.239674 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:02.226617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:33439)
0620 21:46:02.227267 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:02.227306 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:02.228101 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:02.228642 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:02.228648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:02.228995 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:02.229580 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:02.229594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.230509 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.230512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:02.231481 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:02.231491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.231980 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.231984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:02.232164 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:02.232758 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:02.232774 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:02.234539 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:46:02.236373 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:46:02.236380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:46:02.236398 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:46:02.236764 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:46:02.237046 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:46:02.237051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:46:02.237054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:46:02.237262 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:46:02.237645 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:46:02.237651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:46:02.238028 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 21:46:02.238280 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 21:46:02.238670 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":123,"threads_started":1}
I20260620 21:46:02.240849 10188 heartbeater.cc:347] Connected to a master server at 127.1.155.190:46155
I20260620 21:46:02.241142 10188 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
I20260620 21:46:02.241806 10188 heartbeater.cc:511] Master 127.1.155.190:46155 requested a full tablet report, sending...
I20260620 21:46:02.242924 9863 ts_manager.cc:194] Registered new tserver with Master: 753490598e3349f58d1e5b2cdd43eb84 (127.1.155.130:33329)
I20260620 21:46:02.243505 9863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:33439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:46:02.257135 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46155
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:46:02.366675 10193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:02.366916 10193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:02.366988 10193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:02.371202 10193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:02.371254 10193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:02.371345 10193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:46:02.376564 10193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:02.377599 10193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:02.378715 10193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:02.387858 10199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:02.387866 10198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:02.387940 10201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:02.388046 10193 server_base.cc:1034] running on GCE node
I20260620 21:46:02.388504 10193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:02.389125 10193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:02.390318 10193 hybrid_clock.cc:648] HybridClock initialized: now 1781991962390284 us; error 41 us; skew 500 ppm
Jun 20 21:46:02 dist-test-slave-5bvn krb5kdc[9818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:46:02.393108 10193 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:46:02.394109 10193 webserver.cc:460] Webserver started at http://127.1.155.131:37559/ using document root <none> and password file <none>
I20260620 21:46:02.394654 10193 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:02.394742 10193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:02.394977 10193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:02.396925 10193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "a2261e21a4dc43a091361c7854f7dd17"
format_stamp: "Formatted at 2026-06-20 21:46:02 on dist-test-slave-5bvn"
server_key: "70ee6a98cafec9243879803f9920acb9"
server_key_iv: "1718432e92dcc21b346adbb54fe6749e"
server_key_version: "encryptionkey@0"
I20260620 21:46:02.397506 10193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "a2261e21a4dc43a091361c7854f7dd17"
format_stamp: "Formatted at 2026-06-20 21:46:02 on dist-test-slave-5bvn"
server_key: "70ee6a98cafec9243879803f9920acb9"
server_key_iv: "1718432e92dcc21b346adbb54fe6749e"
server_key_version: "encryptionkey@0"
I20260620 21:46:02.401204 10193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 21:46:02.403672 10208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:02.404912 10193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:46:02.405064 10193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "a2261e21a4dc43a091361c7854f7dd17"
format_stamp: "Formatted at 2026-06-20 21:46:02 on dist-test-slave-5bvn"
server_key: "70ee6a98cafec9243879803f9920acb9"
server_key_iv: "1718432e92dcc21b346adbb54fe6749e"
server_key_version: "encryptionkey@0"
I20260620 21:46:02.405176 10193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:02.420929 10193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:02.424101 10193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:02.424301 10193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:02.424870 10193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:46:02.425642 10193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:46:02.425712 10193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:02.425783 10193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:46:02.425825 10193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:02.435434 10193 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:34685
I20260620 21:46:02.435460 10321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:34685 every 8 connection(s)
I20260620 21:46:02.436292 10193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:46:02.443486 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 10193
I20260620 21:46:02.443663 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:46:02.443948 1646 external_mini_cluster.cc:1428] Setting key 5ac440b2e0d4e30e1253aa15b30a8693
Jun 20 21:46:02 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:46:02.449385 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:02.438092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:38017 (local address 127.1.155.190:46155)
0620 21:46:02.438227 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:02.438231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:02.439199 (+ 968us) server_negotiation.cc:407] Connection header received
0620 21:46:02.440094 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:02.440098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:02.440155 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:02.440257 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:02.441384 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.441975 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.443091 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.443343 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.443451 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:02.446318 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:02.446321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:02.446323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:46:02.446344 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:02.447586 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:02.448032 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:02.448034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:02.448035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:02.448107 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:02.448428 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:02.448438 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:02.448440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:02.448616 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:02.448706 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:02.449162 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 21:46:02.449265 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:46:02.450165 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:02.438426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:38017)
0620 21:46:02.439048 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:02.439087 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:02.439898 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:02.440433 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:02.440440 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:02.440783 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:02.441241 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:02.441254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.442093 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.442099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:02.442956 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:02.442965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.443490 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.443494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:02.443709 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:02.444376 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:02.444396 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:02.446173 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:46:02.447687 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:46:02.447692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:46:02.447702 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:46:02.447949 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:46:02.448204 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:46:02.448206 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:46:02.448207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:46:02.448323 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:46:02.448719 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:46:02.448724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:46:02.448940 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 21:46:02.449086 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 21:46:02.449315 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":105,"threads_started":1}
I20260620 21:46:02.451201 10322 heartbeater.cc:347] Connected to a master server at 127.1.155.190:46155
I20260620 21:46:02.451400 10322 heartbeater.cc:464] Registering TS with master...
I20260620 21:46:02.451804 10322 heartbeater.cc:511] Master 127.1.155.190:46155 requested a full tablet report, sending...
I20260620 21:46:02.452903 9863 ts_manager.cc:194] Registered new tserver with Master: a2261e21a4dc43a091361c7854f7dd17 (127.1.155.131:34685)
I20260620 21:46:02.453485 9863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:38017
I20260620 21:46:02.457619 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.129.keytab.
Entry for principal oryx/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.129.keytab.
WARNING: no policy specified for oryx/127.1.155.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.155.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.130.keytab.
Entry for principal oryx/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.130.keytab.
WARNING: no policy specified for oryx/127.1.155.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.155.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.131.keytab.
Entry for principal oryx/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.131.keytab.
I20260620 21:46:02.532411 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 9925
I20260620 21:46:02.537810 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 10059
I20260620 21:46:02.543190 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 10193
I20260620 21:46:02.548422 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 9831
I20260620 21:46:02.554910 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:46155
--webserver_interface=127.1.155.190
--webserver_port=34859
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:46155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:46:02.665194 10331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:02.665503 10331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:02.665576 10331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:02.669716 10331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:46:02.669798 10331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:02.669870 10331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:02.669921 10331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:46:02.669960 10331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:46:02.675146 10331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:46155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:46155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:02.676190 10331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:02.677171 10331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:02.685506 10339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:02.685503 10337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:02.685539 10331 server_base.cc:1034] running on GCE node
W20260620 21:46:02.685503 10336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:02.686327 10331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:02.687393 10331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:02.688589 10331 hybrid_clock.cc:648] HybridClock initialized: now 1781991962688577 us; error 25 us; skew 500 ppm
Jun 20 21:46:02 dist-test-slave-5bvn krb5kdc[9818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:46:02.691502 10331 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:46:02.692399 10331 webserver.cc:460] Webserver started at http://127.1.155.190:34859/ using document root <none> and password file <none>
I20260620 21:46:02.692945 10331 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:02.693046 10331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:02.696750 10331 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:46:02.698925 10346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:02.700127 10331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:46:02.700346 10331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "275e5a201e1d4cf28791dc1537bc48c3"
format_stamp: "Formatted at 2026-06-20 21:46:01 on dist-test-slave-5bvn"
server_key: "6f71877de08c152fbb0c7d58f2030e64"
server_key_iv: "05f033f10ca03fe3e642fae527a1562b"
server_key_version: "encryptionkey@0"
I20260620 21:46:02.700903 10331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:02.713125 10331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:02.716156 10331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:02.716367 10331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:02.723846 10331 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:46155
I20260620 21:46:02.723862 10398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:46155 every 8 connection(s)
I20260620 21:46:02.724901 10331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:46:02.729174 10399 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:46:02.730876 10399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3: Bootstrap starting.
I20260620 21:46:02.731037 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 10331
I20260620 21:46:02.731600 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:40857
--local_ip_for_outbound_sockets=127.1.155.129
--tserver_master_addrs=127.1.155.190:46155
--webserver_port=35347
--webserver_interface=127.1.155.129
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.155.129
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:46:02.747646 10399 log.cc:824] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3: Log is configured to *not* fsync() on all Append() calls
I20260620 21:46:02.754071 10399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:46:02.754431 10399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3: Bootstrap complete.
I20260620 21:46:02.758579 10399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275e5a201e1d4cf28791dc1537bc48c3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46155 } }
I20260620 21:46:02.758967 10399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275e5a201e1d4cf28791dc1537bc48c3, State: Initialized, Role: FOLLOWER
I20260620 21:46:02.759430 10399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275e5a201e1d4cf28791dc1537bc48c3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46155 } }
I20260620 21:46:02.759567 10399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:46:02.759632 10399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:46:02.759748 10399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:46:02.761626 10399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275e5a201e1d4cf28791dc1537bc48c3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46155 } }
I20260620 21:46:02.761981 10399 leader_election.cc:302] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 275e5a201e1d4cf28791dc1537bc48c3; no voters:
I20260620 21:46:02.762305 10399 leader_election.cc:288] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 21:46:02.762387 10403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:46:02.762566 10403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [term 2 LEADER]: Becoming Leader. State: Replica: 275e5a201e1d4cf28791dc1537bc48c3, State: Running, Role: LEADER
I20260620 21:46:02.762843 10403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275e5a201e1d4cf28791dc1537bc48c3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46155 } }
I20260620 21:46:02.763477 10399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:46:02.764096 10404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "275e5a201e1d4cf28791dc1537bc48c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275e5a201e1d4cf28791dc1537bc48c3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46155 } } }
I20260620 21:46:02.764218 10404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [sys.catalog]: This master's current role is: LEADER
I20260620 21:46:02.764809 10405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 275e5a201e1d4cf28791dc1537bc48c3. Latest consensus state: current_term: 2 leader_uuid: "275e5a201e1d4cf28791dc1537bc48c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275e5a201e1d4cf28791dc1537bc48c3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46155 } } }
I20260620 21:46:02.764956 10405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3 [sys.catalog]: This master's current role is: LEADER
I20260620 21:46:02.765157 10410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:46:02.768271 10410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:46:02.769095 10410 catalog_manager.cc:1259] Loaded cluster ID: 6566ac45940543b194fdd01981bd13ea
I20260620 21:46:02.769155 10410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:46:02.771451 10410 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:46:02.772107 10410 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 275e5a201e1d4cf28791dc1537bc48c3: Loaded TSK: 0
I20260620 21:46:02.773262 10410 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:46:02.845625 10401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:02.845943 10401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:02.846014 10401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:02.850059 10401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:02.850149 10401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:02.850244 10401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:46:02.855406 10401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.129.keytab
--principal=oryx/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:40857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=35347
--tserver_master_addrs=127.1.155.190:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:02.856388 10401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:02.857463 10401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:02.866847 10425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:02.866825 10428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:02.866817 10426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:02.867218 10401 server_base.cc:1034] running on GCE node
I20260620 21:46:02.867599 10401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:02.868197 10401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:02.869414 10401 hybrid_clock.cc:648] HybridClock initialized: now 1781991962869384 us; error 45 us; skew 500 ppm
Jun 20 21:46:02 dist-test-slave-5bvn krb5kdc[9818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991962, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:46:02.872735 10401 init.cc:377] Logged in from keytab as oryx/127.1.155.129@KRBTEST.COM (short username oryx)
I20260620 21:46:02.873684 10401 webserver.cc:460] Webserver started at http://127.1.155.129:35347/ using document root <none> and password file <none>
I20260620 21:46:02.874322 10401 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:02.874428 10401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:02.878298 10401 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:46:02.880350 10435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:02.881742 10401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:46:02.881880 10401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "ad13b76e05f84e16bf71e3c602a2e510"
format_stamp: "Formatted at 2026-06-20 21:46:01 on dist-test-slave-5bvn"
server_key: "6bf3e8910b80a900b3082c2fe02b8321"
server_key_iv: "f30663721aee596e6b3f9cd24f238a1c"
server_key_version: "encryptionkey@0"
I20260620 21:46:02.882421 10401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:02.902294 10401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:02.905758 10401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:02.905987 10401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:02.906605 10401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:46:02.907421 10401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:46:02.907466 10401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:02.907536 10401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:46:02.907567 10401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:02.916858 10401 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:40857
I20260620 21:46:02.916890 10548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:40857 every 8 connection(s)
I20260620 21:46:02.917852 10401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:46:02.918254 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 10401
I20260620 21:46:02.918793 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:33329
--local_ip_for_outbound_sockets=127.1.155.130
--tserver_master_addrs=127.1.155.190:46155
--webserver_port=45935
--webserver_interface=127.1.155.130
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.155.130
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:46:02 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:02 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:02.932906 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:02.920091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:45637)
0620 21:46:02.920877 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:02.920927 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:02.921671 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:02.922802 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:02.922813 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:02.923345 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:02.924036 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:02.924058 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.925356 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.925362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:02.926968 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:02.926983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.927889 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.927895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:02.928122 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:02.928829 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:02.928857 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:02.931875 (+ 3018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":126,"threads_started":1}
W20260620 21:46:02.932895 10552 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:02.920030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:45637 (local address 127.1.155.190:46155)
0620 21:46:02.920546 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:02.920570 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:02.921055 (+ 485us) server_negotiation.cc:407] Connection header received
0620 21:46:02.922108 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:02.922119 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:02.922401 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:02.922663 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:02.924223 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.925203 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.927245 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.927707 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.927836 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:02.932105 (+ 4269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:45637: BlockingRecv error: recv got EOF from 127.1.155.129:45637 (error 108)
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":125,"threads_started":1}
W20260620 21:46:02.933478 10549 heartbeater.cc:644] Failed to heartbeat to 127.1.155.190:46155 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.155.190:46155: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Jun 20 21:46:02 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:02 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:02.939591 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:02.933977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:45099)
0620 21:46:02.934136 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:02.934149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:02.934282 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:02.934526 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:02.934529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:02.934687 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:02.934793 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:02.934798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.935678 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.935682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:02.936582 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:02.936592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.937028 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.937031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:02.937122 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:02.937571 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:02.937584 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:02.939466 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260620 21:46:02.939756 10552 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:02.934048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:45099 (local address 127.1.155.190:46155)
0620 21:46:02.934155 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:02.934159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:02.934209 (+ 50us) server_negotiation.cc:407] Connection header received
0620 21:46:02.934374 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:02.934376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:02.934420 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:02.934495 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:02.934916 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.935572 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.936701 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.936924 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.937008 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:02.939604 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:45099: BlockingRecv error: recv got EOF from 127.1.155.129:45099 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 21:46:02 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:02 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:02.945271 10554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:02.940057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:41731)
0620 21:46:02.940377 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:02.940389 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:02.940506 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:02.940771 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:02.940774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:02.940957 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:02.941036 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:02.941041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.941727 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.941731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:02.942370 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:02.942378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.942823 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.942827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:02.942925 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:02.943431 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:02.943443 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:02.945093 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":87,"threads_started":1}
W20260620 21:46:02.945333 10552 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:02.940150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:41731 (local address 127.1.155.190:46155)
0620 21:46:02.940259 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:02.940263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:02.940467 (+ 204us) server_negotiation.cc:407] Connection header received
0620 21:46:02.940622 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:02.940627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:02.940674 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:02.940756 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:02.941146 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.941621 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.942501 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:02.942705 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:02.942817 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:02.945227 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:41731: BlockingRecv error: recv got EOF from 127.1.155.129:41731 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 21:46:02.945608 10549 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:46:03.034420 10551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:03.034664 10551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:03.034706 10551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:03.038823 10551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:03.038882 10551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:03.038985 10551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:46:03.044103 10551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.130.keytab
--principal=oryx/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:33329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=45935
--tserver_master_addrs=127.1.155.190:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:03.045138 10551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:03.046268 10551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:03.055343 10560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:03.055375 10559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:03.055337 10562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:03.055620 10551 server_base.cc:1034] running on GCE node
I20260620 21:46:03.056043 10551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:03.056628 10551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:03.057785 10551 hybrid_clock.cc:648] HybridClock initialized: now 1781991963057771 us; error 25 us; skew 500 ppm
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991963, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:46:03.060367 10551 init.cc:377] Logged in from keytab as oryx/127.1.155.130@KRBTEST.COM (short username oryx)
I20260620 21:46:03.061209 10551 webserver.cc:460] Webserver started at http://127.1.155.130:45935/ using document root <none> and password file <none>
I20260620 21:46:03.061749 10551 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:03.061820 10551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:03.066028 10551 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:46:03.068068 10569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:03.069242 10551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:46:03.069375 10551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "753490598e3349f58d1e5b2cdd43eb84"
format_stamp: "Formatted at 2026-06-20 21:46:02 on dist-test-slave-5bvn"
server_key: "2be90ef94b5eff4096a817d7d73ff638"
server_key_iv: "173a1a50c0d8ca6045dd95e591f33e0a"
server_key_version: "encryptionkey@0"
I20260620 21:46:03.069947 10551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:03.103500 10551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:03.106660 10551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:03.106860 10551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:03.107445 10551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:46:03.108242 10551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:46:03.108311 10551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:03.108382 10551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:46:03.108426 10551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:03.117251 10551 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:33329
I20260620 21:46:03.117277 10682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:33329 every 8 connection(s)
I20260620 21:46:03.118197 10551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:03.126708 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 10551
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:03.127206 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:34685
--local_ip_for_outbound_sockets=127.1.155.131
--tserver_master_addrs=127.1.155.190:46155
--webserver_port=37559
--webserver_interface=127.1.155.131
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.155.131
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:46:03.128368 10552 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:03.119928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:59235 (local address 127.1.155.190:46155)
0620 21:46:03.120057 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:03.120061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:03.120970 (+ 909us) server_negotiation.cc:407] Connection header received
0620 21:46:03.121744 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:03.121746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:03.121796 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:03.121914 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:03.122987 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.123503 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.124566 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.124765 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.124839 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:03.128268 (+ 3429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:59235: BlockingRecv error: recv got EOF from 127.1.155.130:59235 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:46:03.129065 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:03.120220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:59235)
0620 21:46:03.120827 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:03.120864 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:03.121562 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:03.122068 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:03.122075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:03.122406 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:03.122863 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:03.122875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.123602 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.123604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:03.124467 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:03.124475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.124886 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.124889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:03.125015 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:03.125550 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:03.125564 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:03.128150 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":101,"threads_started":1}
W20260620 21:46:03.129667 10683 heartbeater.cc:644] Failed to heartbeat to 127.1.155.190:46155 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.155.190:46155: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:03.135257 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:03.130174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:45337)
0620 21:46:03.130281 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:03.130297 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:03.130371 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:03.130588 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:03.130591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:03.130742 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:03.130824 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:03.130829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.131525 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.131531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:03.132069 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:03.132076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.132517 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.132520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:03.132608 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:03.133157 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:03.133169 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:03.135101 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 21:46:03.135325 10552 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:03.130270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:45337 (local address 127.1.155.190:46155)
0620 21:46:03.130379 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:03.130383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:03.130397 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:46:03.130445 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:03.130448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:03.130490 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:03.130574 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:03.130933 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.131422 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.132184 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.132411 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.132516 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:03.135215 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:45337: BlockingRecv error: recv got EOF from 127.1.155.130:45337 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:03.140822 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:03.135884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:46795)
0620 21:46:03.135981 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:03.135995 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:03.136090 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:03.136308 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:03.136311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:03.136454 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:03.136516 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:03.136521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.137263 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.137269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:03.137780 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:03.137787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.138280 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.138283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:03.138358 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:03.138823 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:03.138836 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:03.140686 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 21:46:03.140908 10552 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:03.135986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:46795 (local address 127.1.155.190:46155)
0620 21:46:03.136110 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:03.136116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:03.136131 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:46:03.136167 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:03.136170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:03.136205 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:03.136294 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:03.136620 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.137152 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.137953 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.138172 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.138280 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:03.140800 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:46795: BlockingRecv error: recv got EOF from 127.1.155.130:46795 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 21:46:03.141148 10683 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:46:03.239038 10686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:03.239305 10686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:03.239347 10686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:03.243382 10686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:03.243443 10686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:03.243549 10686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:46:03.248761 10686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/krb5kdc/oryx_127.1.155.131.keytab
--principal=oryx/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:34685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=37559
--tserver_master_addrs=127.1.155.190:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:03.249766 10686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:03.250764 10686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:03.259966 10691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:03.259984 10692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:03.259943 10694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:03.260314 10686 server_base.cc:1034] running on GCE node
I20260620 21:46:03.260675 10686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:03.261250 10686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:03.262401 10686 hybrid_clock.cc:648] HybridClock initialized: now 1781991963262383 us; error 27 us; skew 500 ppm
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991963, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:46:03.265465 10686 init.cc:377] Logged in from keytab as oryx/127.1.155.131@KRBTEST.COM (short username oryx)
I20260620 21:46:03.266366 10686 webserver.cc:460] Webserver started at http://127.1.155.131:37559/ using document root <none> and password file <none>
I20260620 21:46:03.266903 10686 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:03.266973 10686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:03.270944 10686 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:46:03.273066 10701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:03.274721 10686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:46:03.274868 10686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "a2261e21a4dc43a091361c7854f7dd17"
format_stamp: "Formatted at 2026-06-20 21:46:02 on dist-test-slave-5bvn"
server_key: "70ee6a98cafec9243879803f9920acb9"
server_key_iv: "1718432e92dcc21b346adbb54fe6749e"
server_key_version: "encryptionkey@0"
I20260620 21:46:03.275707 10686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:03.293450 10686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:03.296728 10686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:03.296965 10686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:03.297596 10686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:46:03.298568 10686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:46:03.298648 10686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:03.298727 10686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:46:03.298785 10686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:03.308805 10686 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:34685
I20260620 21:46:03.308828 10814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:34685 every 8 connection(s)
I20260620 21:46:03.309826 10686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:46:03.314572 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 10686
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 21:46:03.320803 10552 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:03.311701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:45341 (local address 127.1.155.190:46155)
0620 21:46:03.311860 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:03.311864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:03.312826 (+ 962us) server_negotiation.cc:407] Connection header received
0620 21:46:03.313649 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:03.313652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:03.313698 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:03.313778 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:03.315306 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.315979 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.317113 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.317329 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.317442 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:03.320686 (+ 3244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:45341: BlockingRecv error: recv got EOF from 127.1.155.131:45341 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 21:46:03.321476 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:03.312057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:45341)
0620 21:46:03.312688 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:03.312726 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:03.313448 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:03.314075 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:03.314082 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:03.314473 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:03.315124 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:03.315148 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.316113 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.316117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:03.316982 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:03.316992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.317478 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.317483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:03.317647 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:03.318240 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:03.318256 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:03.320583 (+ 2327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":117,"threads_started":1}
W20260620 21:46:03.322017 10815 heartbeater.cc:644] Failed to heartbeat to 127.1.155.190:46155 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.155.190:46155: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
I20260620 21:46:03.324694 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:03.315623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.1.155.190:46155)
0620 21:46:03.315875 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:03.315878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:03.315890 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:46:03.315922 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:03.315924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:03.315960 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:03.316073 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:03.316833 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.317356 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.318324 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.318565 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.318699 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:03.320216 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:03.320223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:03.320250 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:46:03.320279 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:03.322460 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:03.322969 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:03.322976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:03.322982 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:03.323111 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:03.323423 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:03.323427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:03.323429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:03.323779 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:03.323989 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:03.324321 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 21:46:03.324560 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:03.327323 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:03.322456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:38047)
0620 21:46:03.322540 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:03.322553 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:03.322623 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:03.322838 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:03.322841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:03.322982 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:03.323060 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:03.323065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.323751 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.323756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:03.324280 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:03.324287 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.324776 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.324779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:03.324870 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:03.325307 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:03.325318 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:03.327233 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 21:46:03.327488 10552 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:03.322534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:38047 (local address 127.1.155.190:46155)
0620 21:46:03.322633 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:03.322637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:03.322651 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:46:03.322696 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:03.322699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:03.322733 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:03.322825 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:03.323165 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.323641 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.324394 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.324670 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.324776 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:03.327389 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:38047: BlockingRecv error: recv got EOF from 127.1.155.131:38047 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:03.332557 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:03.327945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:44681)
0620 21:46:03.328098 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:03.328113 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:03.328185 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:03.328408 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:03.328411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:03.328560 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:03.328621 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:03.328625 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.329293 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.329299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:03.329808 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:03.329815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.330296 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.330300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:03.330375 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:03.330860 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:03.330870 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:03.332424 (+ 1554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 21:46:03.332631 10552 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:03.328035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:44681 (local address 127.1.155.190:46155)
0620 21:46:03.328179 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:03.328183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:03.328196 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:46:03.328275 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:03.328279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:03.328316 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:03.328397 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:03.328728 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.329191 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.329952 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.330159 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.330287 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:03.332522 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:44681: BlockingRecv error: recv got EOF from 127.1.155.131:44681 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 21:46:03.332839 10815 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:03 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:03.952138 10820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:03.946341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:34659)
0620 21:46:03.946617 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:03.946631 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:03.946719 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:03.946967 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:03.946969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:03.947151 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:03.947266 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:03.947271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.948358 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.948361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:03.949006 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:03.949016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.949444 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.949446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:03.949517 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:03.950028 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:03.950039 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:03.951996 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
W20260620 21:46:03.952416 10821 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:03.946421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:34659 (local address 127.1.155.190:46155)
0620 21:46:03.946691 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:03.946694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:03.946709 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:46:03.946807 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:03.946809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:03.946852 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:03.946933 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:03.947390 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.948227 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.949117 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:03.949339 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:03.949417 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:03.952252 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:34659: BlockingRecv error: recv got EOF from 127.1.155.129:34659 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":108,"threads_started":1}
Jun 20 21:46:04 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:04 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:04.147796 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:04.141783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:59123)
0620 21:46:04.142124 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:04.142144 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:04.142253 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:04.142548 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:04.142551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:04.142804 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:04.142921 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:04.142929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:04.143825 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:04.143828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:04.144426 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:04.144435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:04.144970 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:04.144973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:04.145036 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:04.145606 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:04.145617 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:04.147675 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":97,"threads_started":1}
W20260620 21:46:04.147939 10821 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:04.142166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:59123 (local address 127.1.155.190:46155)
0620 21:46:04.142319 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:04.142324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:04.142339 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:46:04.142376 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:04.142379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:04.142428 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:04.142527 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:04.143074 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:04.143699 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:04.144583 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:04.144848 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:04.144968 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:04.147818 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:59123: BlockingRecv error: recv got EOF from 127.1.155.130:59123 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 21:46:04 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:04 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:04.339632 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:04.333573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:43053)
0620 21:46:04.333886 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:04.333900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:04.334000 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:04.334356 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:04.334359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:04.334538 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:04.334651 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:04.334657 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:04.335528 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:04.335531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:04.336104 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:04.336112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:04.336588 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:04.336591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:04.336654 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:04.337244 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:04.337261 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:04.339510 (+ 2249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":97,"threads_started":1}
W20260620 21:46:04.339831 10821 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:04.333674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:43053 (local address 127.1.155.190:46155)
0620 21:46:04.333886 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:04.333891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:04.333952 (+ 61us) server_negotiation.cc:407] Connection header received
0620 21:46:04.334125 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:04.334128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:04.334187 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:04.334266 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:04.334842 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:04.335423 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:04.336232 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:04.336467 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:04.336558 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:04.339653 (+ 3095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:43053: BlockingRecv error: recv got EOF from 127.1.155.131:43053 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 21:46:04 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:04 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:04.958788 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:04.952927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:53839)
0620 21:46:04.953211 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:04.953224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:04.953309 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:04.953569 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:04.953571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:04.953741 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:04.953898 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:04.953909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:04.954697 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:04.954699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:04.955265 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:04.955274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:04.955747 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:04.955750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:04.955825 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:04.956352 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:04.956364 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:04.958656 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":118,"threads_started":1}
W20260620 21:46:04.958940 10825 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:04.953114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:53839 (local address 127.1.155.190:46155)
0620 21:46:04.953360 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:04.953364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:04.953375 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:46:04.953408 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:04.953410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:04.953448 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:04.953532 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:04.954037 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:04.954568 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:04.955382 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:04.955621 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:04.955704 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:04.958798 (+ 3094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:53839: BlockingRecv error: recv got EOF from 127.1.155.129:53839 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
Jun 20 21:46:05 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:05 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:05.154275 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:05.148818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:48543)
0620 21:46:05.149086 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:05.149101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:05.149187 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:05.149412 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:05.149414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:05.149587 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:05.149693 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:05.149698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:05.150456 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:05.150459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:05.151060 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:05.151069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:05.151508 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:05.151511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:05.151572 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:05.152070 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:05.152081 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:05.154137 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
W20260620 21:46:05.154420 10825 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:05.149036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:48543 (local address 127.1.155.190:46155)
0620 21:46:05.149180 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:05.149183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:05.149196 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:46:05.149270 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:05.149272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:05.149314 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:05.149375 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:05.149801 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:05.150354 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:05.151168 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:05.151395 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:05.151500 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:05.154292 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:48543: BlockingRecv error: recv got EOF from 127.1.155.130:48543 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 21:46:05 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:05 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:05.346374 10827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:05.340583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:51125)
0620 21:46:05.340869 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:05.340898 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:05.341003 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:05.341311 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:05.341314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:05.341536 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:05.341631 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:05.341637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:05.342596 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:05.342599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:05.343151 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:05.343159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:05.343631 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:05.343634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:05.343702 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:05.344289 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:05.344300 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:05.346185 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":115,"threads_started":1}
W20260620 21:46:05.346414 10825 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:05.340781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:51125 (local address 127.1.155.190:46155)
0620 21:46:05.340962 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:05.340966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:05.340983 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:46:05.341079 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:05.341081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:05.341129 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:05.341216 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:05.341951 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:05.342475 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:05.343273 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:05.343497 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:05.343589 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:05.346304 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:51125: BlockingRecv error: recv got EOF from 127.1.155.131:51125 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 20 21:46:05 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:05 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:05.969121 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:05.959645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:42219)
0620 21:46:05.959982 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:05.959996 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:05.960090 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:05.960347 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:05.960349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:05.960512 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:05.960601 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:05.960606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:05.961393 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:05.961396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:05.961988 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:05.961997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:05.962459 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:05.962462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:05.962526 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:05.967155 (+ 4629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:05.967170 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:05.969000 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
W20260620 21:46:05.969367 10829 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:05.959771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:42219 (local address 127.1.155.190:46155)
0620 21:46:05.960029 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:05.960033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:05.960046 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:46:05.960203 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:05.960206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:05.960252 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:05.960336 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:05.960707 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:05.961253 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:05.962098 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:05.962329 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:05.962431 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:05.969203 (+ 6772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:42219: BlockingRecv error: recv got EOF from 127.1.155.129:42219 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
Jun 20 21:46:06 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:06 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:06.161074 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:06.155089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:58217)
0620 21:46:06.155367 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:06.155452 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:06.155606 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:06.155873 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:06.155876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:06.156078 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:06.156164 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:06.156170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:06.156929 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:06.156935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:06.157495 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:06.157503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:06.158002 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:06.158006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:06.158092 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:06.158707 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:06.158744 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:06.160864 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":116,"threads_started":1}
W20260620 21:46:06.161092 10829 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:06.155426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:58217 (local address 127.1.155.190:46155)
0620 21:46:06.155593 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:06.155597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:06.155614 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:46:06.155702 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:06.155705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:06.155756 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:06.155845 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:06.156281 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:06.156812 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:06.157614 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:06.157889 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:06.157997 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:06.160995 (+ 2998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:58217: BlockingRecv error: recv got EOF from 127.1.155.130:58217 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 21:46:06 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:06 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:06.353080 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:06.347264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:52227)
0620 21:46:06.347551 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:06.347565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:06.347659 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:06.347920 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:06.347922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:06.348094 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:06.348185 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:06.348190 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:06.348936 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:06.348940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:06.349638 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:06.349650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:06.350242 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:06.350246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:06.350344 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:06.350953 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:06.350974 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:06.352966 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":116,"threads_started":1}
W20260620 21:46:06.353271 10829 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:06.347430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:52227 (local address 127.1.155.190:46155)
0620 21:46:06.347577 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:06.347581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:06.347627 (+ 46us) server_negotiation.cc:407] Connection header received
0620 21:46:06.347745 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:06.347747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:06.347792 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:06.347869 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:06.348285 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:06.348807 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:06.349752 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:06.350110 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:06.350226 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:06.353104 (+ 2878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:52227: BlockingRecv error: recv got EOF from 127.1.155.131:52227 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 21:46:06 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:06 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:06.977100 10832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:06.970008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:51465)
0620 21:46:06.970384 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:06.970398 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:06.970507 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:06.970882 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:06.970893 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:06.971074 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:06.971167 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:06.971173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:06.972294 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:06.972296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:06.973365 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:06.973376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:06.974017 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:06.974022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:06.974120 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:06.974691 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:06.974712 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:06.976980 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":98,"threads_started":1}
W20260620 21:46:06.977324 10833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:06.970146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:51465 (local address 127.1.155.190:46155)
0620 21:46:06.970411 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:06.970415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:06.970484 (+ 69us) server_negotiation.cc:407] Connection header received
0620 21:46:06.970663 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:06.970667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:06.970719 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:06.970814 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:06.971305 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:06.972150 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:06.973616 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:06.973860 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:06.973974 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:06.977144 (+ 3170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:51465: BlockingRecv error: recv got EOF from 127.1.155.129:51465 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":58,"threads_started":1}
Jun 20 21:46:07 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:07 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:07.167877 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:07.162141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:55729)
0620 21:46:07.162467 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:07.162491 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:07.162630 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:07.162940 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:07.162943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:07.163147 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:07.163237 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:07.163242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:07.164080 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:07.164086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:07.164749 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:07.164758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:07.165235 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:07.165239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:07.165324 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:07.165909 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:07.165922 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:07.167762 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":78,"threads_started":1}
W20260620 21:46:07.168166 10833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:07.162266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:55729 (local address 127.1.155.190:46155)
0620 21:46:07.162447 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:07.162452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:07.162567 (+ 115us) server_negotiation.cc:407] Connection header received
0620 21:46:07.162767 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:07.162771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:07.162825 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:07.162917 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:07.163354 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:07.163971 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:07.164866 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:07.165088 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:07.165221 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:07.167937 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:55729: BlockingRecv error: recv got EOF from 127.1.155.130:55729 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 21:46:07 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:07 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:07.359941 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:07.353933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:49411)
0620 21:46:07.354278 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:07.354296 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:07.354420 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:07.354712 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:07.354716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:07.354949 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:07.355042 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:07.355048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:07.355816 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:07.355821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:07.356379 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:07.356386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:07.356841 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:07.356845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:07.356931 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:07.357449 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:07.357460 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:07.359690 (+ 2230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":133,"threads_started":1}
W20260620 21:46:07.360015 10833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:07.354114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:49411 (local address 127.1.155.190:46155)
0620 21:46:07.354288 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:07.354295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:07.354362 (+ 67us) server_negotiation.cc:407] Connection header received
0620 21:46:07.354537 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:07.354541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:07.354598 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:07.354688 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:07.355165 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:07.355695 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:07.356501 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:07.356722 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:07.356839 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:07.359892 (+ 3053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:49411: BlockingRecv error: recv got EOF from 127.1.155.131:49411 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 21:46:07 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:07 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:07.984309 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:07.978104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:43579)
0620 21:46:07.978538 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:07.978579 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:07.978702 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:07.979093 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:07.979095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:07.979368 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:07.979504 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:07.979512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:07.980380 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:07.980384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:07.980953 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:07.980960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:07.981488 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:07.981491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:07.981575 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:07.982246 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:07.982258 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:07.984171 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":93,"threads_started":1}
W20260620 21:46:07.984468 10837 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:07.978215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:43579 (local address 127.1.155.190:46155)
0620 21:46:07.978578 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:07.978583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:07.978677 (+ 94us) server_negotiation.cc:407] Connection header received
0620 21:46:07.978855 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:07.978860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:07.978921 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:07.979058 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:07.979621 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:07.980263 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:07.981073 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:07.981375 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:07.981487 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:07.984303 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:43579: BlockingRecv error: recv got EOF from 127.1.155.129:43579 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":89,"threads_started":1}
Jun 20 21:46:08 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:08 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:08.173985 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:08.168658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:57431)
0620 21:46:08.168926 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:08.168939 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:08.169019 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:08.169243 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:08.169246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:08.169417 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:08.169508 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:08.169513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:08.170349 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:08.170352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:08.170936 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:08.170945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:08.171378 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:08.171380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:08.171441 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:08.171913 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:08.171924 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:08.173796 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":98,"threads_started":1}
W20260620 21:46:08.174099 10837 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:08.168816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:57431 (local address 127.1.155.190:46155)
0620 21:46:08.168958 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:08.168961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:08.168977 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:46:08.169098 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:08.169100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:08.169138 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:08.169206 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:08.169604 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:08.170192 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:08.171055 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:08.171279 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:08.171361 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:08.173969 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:57431: BlockingRecv error: recv got EOF from 127.1.155.130:57431 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 21:46:08 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:08 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:08.366731 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:08.360985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:57043)
0620 21:46:08.361258 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:08.361273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:08.361405 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:08.361692 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:08.361695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:08.361968 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:08.362055 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:08.362061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:08.362803 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:08.362810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:08.363372 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:08.363379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:08.363914 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:08.363918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:08.364002 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:08.364570 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:08.364581 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:08.366533 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":84,"threads_started":1}
W20260620 21:46:08.366760 10837 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:08.361054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:57043 (local address 127.1.155.190:46155)
0620 21:46:08.361203 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:08.361209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:08.361359 (+ 150us) server_negotiation.cc:407] Connection header received
0620 21:46:08.361501 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:08.361504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:08.361558 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:08.361646 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:08.362175 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:08.362694 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:08.363493 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:08.363796 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:08.363911 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:08.366658 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:57043: BlockingRecv error: recv got EOF from 127.1.155.131:57043 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 21:46:08 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:08 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:08.990597 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:08.985108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:36093)
0620 21:46:08.985400 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:08.985425 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:08.985581 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:08.985950 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:08.985953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:08.986160 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:08.986251 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:08.986257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:08.987039 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:08.987042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:08.987614 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:08.987622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:08.988062 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:08.988065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:08.988120 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:08.988660 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:08.988673 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:08.990484 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
W20260620 21:46:08.990907 10841 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:08.985183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:36093 (local address 127.1.155.190:46155)
0620 21:46:08.985429 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:08.985433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:08.985662 (+ 229us) server_negotiation.cc:407] Connection header received
0620 21:46:08.985705 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:08.985708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:08.985770 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:08.985928 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:08.986357 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:08.986942 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:08.987743 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:08.987965 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:08.988051 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:08.990625 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:36093: BlockingRecv error: recv got EOF from 127.1.155.129:36093 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":55,"threads_started":1}
Jun 20 21:46:09 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:09 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:09.180379 10842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:09.174786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:34007)
0620 21:46:09.175123 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:09.175138 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:09.175279 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:09.175553 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:09.175556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:09.175780 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:09.175868 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:09.175874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:09.176643 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:09.176649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:09.177223 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:09.177231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:09.177686 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:09.177689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:09.177772 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:09.178347 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:09.178359 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:09.180266 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":83,"threads_started":1}
W20260620 21:46:09.180511 10841 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:09.174923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:34007 (local address 127.1.155.190:46155)
0620 21:46:09.175093 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:09.175098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:09.175226 (+ 128us) server_negotiation.cc:407] Connection header received
0620 21:46:09.175385 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:09.175389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:09.175443 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:09.175533 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:09.175989 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:09.176506 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:09.177344 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:09.177573 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:09.177684 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:09.180394 (+ 2710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:34007: BlockingRecv error: recv got EOF from 127.1.155.130:34007 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 21:46:09 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:09 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:09.373116 10843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:09.367530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:43871)
0620 21:46:09.367810 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:09.367822 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:09.367917 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:09.368172 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:09.368175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:09.368350 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:09.368437 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:09.368442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:09.369223 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:09.369226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:09.369813 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:09.369821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:09.370299 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:09.370302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:09.370366 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:09.370940 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:09.370951 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:09.372917 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":108,"threads_started":1}
W20260620 21:46:09.373239 10841 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:09.367602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:43871 (local address 127.1.155.190:46155)
0620 21:46:09.367765 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:09.367769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:09.367895 (+ 126us) server_negotiation.cc:407] Connection header received
0620 21:46:09.368012 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:09.368015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:09.368061 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:09.368138 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:09.368541 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:09.369117 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:09.369972 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:09.370190 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:09.370277 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:09.373071 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:43871: BlockingRecv error: recv got EOF from 127.1.155.131:43871 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 21:46:09 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:09 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:09.997730 10844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:09.991413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:42597)
0620 21:46:09.991727 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:09.991740 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:09.991827 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:09.992090 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:09.992092 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:09.992281 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:09.992372 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:09.992377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:09.993269 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:09.993272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:09.993896 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:09.993905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:09.994409 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:09.994412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:09.994486 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:09.995260 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:09.995278 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:09.997546 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":87,"threads_started":1}
W20260620 21:46:09.997802 10845 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:09.991515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:42597 (local address 127.1.155.190:46155)
0620 21:46:09.991759 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:09.991763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:09.991780 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:46:09.991948 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:09.991951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:09.991999 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:09.992081 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:09.992628 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:09.993160 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:09.994031 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:09.994259 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:09.994342 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:09.997677 (+ 3335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:42597: BlockingRecv error: recv got EOF from 127.1.155.129:42597 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":90,"threads_started":1}
Jun 20 21:46:10 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:10 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:10.186604 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:10.181276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:49747)
0620 21:46:10.181629 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:10.181642 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:10.181738 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:10.181980 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:10.181982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:10.182155 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:10.182244 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:10.182248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:10.182999 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:10.183002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:10.183552 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:10.183560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:10.183954 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:10.183957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:10.184015 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:10.184505 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:10.184517 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:10.186434 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":92,"threads_started":1}
W20260620 21:46:10.186663 10845 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:10.181327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:49747 (local address 127.1.155.190:46155)
0620 21:46:10.181523 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:10.181528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:10.181718 (+ 190us) server_negotiation.cc:407] Connection header received
0620 21:46:10.181823 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:10.181825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:10.181891 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:10.181961 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:10.182366 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:10.182878 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:10.183656 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:10.183862 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:10.183937 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:10.186560 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:49747: BlockingRecv error: recv got EOF from 127.1.155.130:49747 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 21:46:10 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:10 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:10.379501 10847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:10.373987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:32861)
0620 21:46:10.374408 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:10.374435 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:10.374539 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:10.374860 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:10.374865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:10.375122 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:10.375254 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:10.375263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:10.376019 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:10.376021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:10.376565 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:10.376572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:10.376964 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:10.376967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:10.377023 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:10.377474 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:10.377485 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:10.379327 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":73,"threads_started":1}
W20260620 21:46:10.379552 10845 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:10.374045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:32861 (local address 127.1.155.190:46155)
0620 21:46:10.374248 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:10.374253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:10.374525 (+ 272us) server_negotiation.cc:407] Connection header received
0620 21:46:10.374660 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:10.374663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:10.374723 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:10.374805 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:10.375374 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:10.375926 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:10.376662 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:10.376867 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:10.376940 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:10.379438 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:32861: BlockingRecv error: recv got EOF from 127.1.155.131:32861 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Jun 20 21:46:11 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:11 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:11.009361 10848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:10.998619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:33507)
0620 21:46:10.999061 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:10.999082 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:10.999330 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:10.999728 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:10.999731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:10.999918 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:11.000025 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:11.000031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:11.000808 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:11.000811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:11.001414 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:11.001423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:11.002115 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:11.002118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:11.002184 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:11.006767 (+ 4583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:11.006779 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:11.009186 (+ 2407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":117,"threads_started":1}
W20260620 21:46:11.009505 10849 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:10.998799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:33507 (local address 127.1.155.190:46155)
0620 21:46:10.999065 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:10.999069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:10.999262 (+ 193us) server_negotiation.cc:407] Connection header received
0620 21:46:10.999435 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:10.999437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:10.999507 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:10.999678 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:11.000147 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:11.000699 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:11.001539 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:11.001781 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:11.002061 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:11.009372 (+ 7311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:33507: BlockingRecv error: recv got EOF from 127.1.155.129:33507 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
Jun 20 21:46:11 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:11 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:11.192930 10850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:11.187404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:55907)
0620 21:46:11.187685 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:11.187698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:11.187806 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:11.188043 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:11.188045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:11.188213 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:11.188333 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:11.188343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:11.189325 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:11.189328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:11.189985 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:11.189995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:11.190440 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:11.190443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:11.190509 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:11.191076 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:11.191088 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:11.192811 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":120,"threads_started":1}
W20260620 21:46:11.193115 10849 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:11.187430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:55907 (local address 127.1.155.190:46155)
0620 21:46:11.187560 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:11.187564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:11.187761 (+ 197us) server_negotiation.cc:407] Connection header received
0620 21:46:11.187896 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:11.187898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:11.187940 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:11.188008 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:11.188459 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:11.189204 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:11.190105 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:11.190320 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:11.190412 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:11.192953 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:55907: BlockingRecv error: recv got EOF from 127.1.155.130:55907 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 21:46:11 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:11 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:11.386037 10851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:11.380481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:60441)
0620 21:46:11.380715 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:11.380733 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:11.380868 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:11.381255 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:11.381258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:11.381504 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:11.381633 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:11.381642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:11.382442 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:11.382444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:11.383006 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:11.383014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:11.383426 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:11.383429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:11.383495 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:11.383975 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:11.383987 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:11.385868 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":77,"threads_started":1}
W20260620 21:46:11.386126 10849 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:11.380515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:60441 (local address 127.1.155.190:46155)
0620 21:46:11.380660 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:11.380664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:11.380794 (+ 130us) server_negotiation.cc:407] Connection header received
0620 21:46:11.380966 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:11.380968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:11.381039 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:11.381128 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:11.381759 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:11.382336 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:11.383102 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:11.383306 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:11.383400 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:11.386011 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:60441: BlockingRecv error: recv got EOF from 127.1.155.131:60441 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 21:46:12 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:12 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:12.015803 10852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:12.010161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:54473)
0620 21:46:12.010433 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:12.010446 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:12.010535 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:12.010778 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:12.010781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:12.010951 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:12.011046 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:12.011051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:12.011835 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:12.011838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:12.012456 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:12.012466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:12.012936 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:12.012938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:12.013013 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:12.013553 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:12.013566 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:12.015579 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":107,"threads_started":1}
W20260620 21:46:12.016091 10853 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:12.010283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:54473 (local address 127.1.155.190:46155)
0620 21:46:12.010504 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:12.010507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:12.010520 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:46:12.010617 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:12.010619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:12.010660 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:12.010741 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:12.011180 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:12.011716 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:12.012572 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:12.012815 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:12.012899 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:12.015754 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:54473: BlockingRecv error: recv got EOF from 127.1.155.129:54473 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
Jun 20 21:46:12 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:12 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:12.198972 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:12.193712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:56787)
0620 21:46:12.193990 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:12.194004 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:12.194099 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:12.194419 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:12.194422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:12.194597 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:12.194689 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:12.194694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:12.195426 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:12.195429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:12.195994 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:12.196002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:12.196427 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:12.196429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:12.196489 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:12.196943 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:12.196954 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:12.198838 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
W20260620 21:46:12.199108 10853 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:12.193777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:56787 (local address 127.1.155.190:46155)
0620 21:46:12.193992 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:12.193996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:12.194054 (+ 58us) server_negotiation.cc:407] Connection header received
0620 21:46:12.194186 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:12.194188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:12.194276 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:12.194343 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:12.194799 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:12.195333 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:12.196094 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:12.196327 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:12.196404 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:12.198962 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:56787: BlockingRecv error: recv got EOF from 127.1.155.130:56787 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Jun 20 21:46:12 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:12 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:12.392520 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:12.386933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:36029)
0620 21:46:12.387189 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:12.387207 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:12.387316 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:12.387598 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:12.387601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:12.387809 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:12.387897 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:12.387903 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:12.388681 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:12.388689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:12.389239 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:12.389246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:12.389690 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:12.389693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:12.389774 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:12.390304 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:12.390315 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:12.392326 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":90,"threads_started":1}
W20260620 21:46:12.392545 10853 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:12.387052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:36029 (local address 127.1.155.190:46155)
0620 21:46:12.387254 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:12.387258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:12.387276 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:46:12.387426 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:12.387430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:12.387483 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:12.387569 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:12.388012 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:12.388564 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:12.389356 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:12.389577 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:12.389687 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:12.392445 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:36029: BlockingRecv error: recv got EOF from 127.1.155.131:36029 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Jun 20 21:46:13 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:13 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:13.022305 10856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:13.016664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:45511)
0620 21:46:13.016964 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:13.016978 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:13.017067 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:13.017369 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:13.017371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:13.017546 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:13.017639 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:13.017645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:13.018474 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:13.018477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:13.019036 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:13.019043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:13.019517 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:13.019519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:13.019576 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:13.020065 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:13.020076 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:13.022186 (+ 2110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":117,"threads_started":1}
W20260620 21:46:13.022486 10857 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:13.016790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:45511 (local address 127.1.155.190:46155)
0620 21:46:13.017021 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:13.017024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:13.017038 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:46:13.017172 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:13.017174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:13.017214 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:13.017297 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:13.017761 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:13.018373 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:13.019162 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:13.019409 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:13.019503 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:13.022338 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:45511: BlockingRecv error: recv got EOF from 127.1.155.129:45511 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
Jun 20 21:46:13 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:13 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:13.205436 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:13.199864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:60237)
0620 21:46:13.200279 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:13.200296 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:13.200424 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:13.200701 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:13.200704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:13.200946 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:13.201068 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:13.201075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:13.201811 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:13.201814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:13.202396 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:13.202405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:13.202848 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:13.202850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:13.202908 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:13.203393 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:13.203405 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:13.205323 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":86,"threads_started":1}
W20260620 21:46:13.205665 10857 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:13.199974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:60237 (local address 127.1.155.190:46155)
0620 21:46:13.200116 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:13.200120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:13.200387 (+ 267us) server_negotiation.cc:407] Connection header received
0620 21:46:13.200500 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:13.200503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:13.200562 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:13.200637 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:13.201187 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:13.201709 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:13.202530 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:13.202748 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:13.202835 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:13.205510 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:60237: BlockingRecv error: recv got EOF from 127.1.155.130:60237 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 21:46:13 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:13 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:13.398942 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:13.393391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:53765)
0620 21:46:13.393723 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:13.393745 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:13.393867 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:13.394098 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:13.394100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:13.394282 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:13.394377 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:13.394382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:13.395120 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:13.395122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:13.395680 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:13.395686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:13.396099 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:13.396101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:13.396183 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:13.396648 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:13.396658 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:13.398779 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
W20260620 21:46:13.398978 10857 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:13.393597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:53765 (local address 127.1.155.190:46155)
0620 21:46:13.393737 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:13.393741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:13.393796 (+ 55us) server_negotiation.cc:407] Connection header received
0620 21:46:13.393949 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:13.393951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:13.393993 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:13.394059 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:13.394512 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:13.395025 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:13.395781 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:13.395995 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:13.396082 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:13.398883 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:53765: BlockingRecv error: recv got EOF from 127.1.155.131:53765 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 21:46:14 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:14 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:14.028867 10860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:14.023255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:41437)
0620 21:46:14.023596 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:14.023610 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:14.023714 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:14.023979 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:14.023981 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:14.024158 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:14.024252 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:14.024257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:14.024978 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:14.024981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:14.025578 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:14.025587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:14.026256 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:14.026258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:14.026319 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:14.026955 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:14.026976 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:14.028752 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":87,"threads_started":1}
W20260620 21:46:14.029004 10861 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:14.023388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:41437 (local address 127.1.155.190:46155)
0620 21:46:14.023630 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:14.023633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:14.023664 (+ 31us) server_negotiation.cc:407] Connection header received
0620 21:46:14.023812 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:14.023815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:14.023859 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:14.023940 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:14.024362 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:14.024883 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:14.025736 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:14.026136 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:14.026244 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:14.028879 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:41437: BlockingRecv error: recv got EOF from 127.1.155.129:41437 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":68,"threads_started":1}
Jun 20 21:46:14 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:14 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:14.211433 10862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:14.206224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:44081)
0620 21:46:14.206477 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:14.206491 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:14.206603 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:14.206831 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:14.206833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:14.207007 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:14.207097 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:14.207102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:14.207859 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:14.207865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:14.208482 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:14.208491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:14.208976 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:14.208979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:14.209043 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:14.209494 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:14.209505 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:14.211323 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
W20260620 21:46:14.211598 10861 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:14.206345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:44081 (local address 127.1.155.190:46155)
0620 21:46:14.206501 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:14.206504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:14.206571 (+ 67us) server_negotiation.cc:407] Connection header received
0620 21:46:14.206686 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:14.206688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:14.206730 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:14.206798 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:14.207191 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:14.207712 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:14.208607 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:14.208826 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:14.208938 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:14.211466 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:44081: BlockingRecv error: recv got EOF from 127.1.155.130:44081 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 21:46:14 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:14 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:14.405503 10863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:14.399697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:35525)
0620 21:46:14.399997 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:14.400010 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:14.400109 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:14.400336 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:14.400339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:14.400516 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:14.400632 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:14.400640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:14.401676 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:14.401679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:14.402354 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:14.402363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:14.402820 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:14.402823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:14.402901 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:14.403391 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:14.403404 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:14.405338 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":82,"threads_started":1}
W20260620 21:46:14.405550 10861 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:14.399779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:35525 (local address 127.1.155.190:46155)
0620 21:46:14.399919 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:14.399922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:14.400082 (+ 160us) server_negotiation.cc:407] Connection header received
0620 21:46:14.400176 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:14.400179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:14.400222 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:14.400296 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:14.400777 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:14.401565 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:14.402464 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:14.402683 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:14.402766 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:14.405443 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:35525: BlockingRecv error: recv got EOF from 127.1.155.131:35525 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 21:46:15 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:15 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:15.035888 10864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:15.029682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:46473)
0620 21:46:15.030068 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:15.030082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:15.030194 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:15.030478 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:15.030480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:15.030656 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:15.030749 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:15.030754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:15.031520 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:15.031523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:15.032238 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:15.032250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:15.032878 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:15.032885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:15.032966 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:15.033608 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:15.033625 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:15.035712 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":143,"threads_started":1}
W20260620 21:46:15.035969 10865 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:15.029869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:46473 (local address 127.1.155.190:46155)
0620 21:46:15.030101 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:15.030104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:15.030163 (+ 59us) server_negotiation.cc:407] Connection header received
0620 21:46:15.030279 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:15.030281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:15.030324 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:15.030403 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:15.030861 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:15.031397 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:15.032395 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:15.032698 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:15.032801 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:15.035844 (+ 3043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:46473: BlockingRecv error: recv got EOF from 127.1.155.129:46473 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":61,"threads_started":1}
Jun 20 21:46:15 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:15 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:15.217783 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:15.212332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:59431)
0620 21:46:15.212603 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:15.212616 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:15.212709 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:15.212991 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:15.212993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:15.213164 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:15.213256 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:15.213261 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:15.214141 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:15.214144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:15.214705 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:15.214722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:15.215246 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:15.215249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:15.215314 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:15.215901 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:15.215914 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:15.217670 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
W20260620 21:46:15.218001 10865 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:15.212427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:59431 (local address 127.1.155.190:46155)
0620 21:46:15.212619 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:15.212622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:15.212667 (+ 45us) server_negotiation.cc:407] Connection header received
0620 21:46:15.212821 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:15.212823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:15.212871 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:15.212948 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:15.213393 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:15.214019 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:15.214856 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:15.215109 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:15.215198 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:15.217810 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:59431: BlockingRecv error: recv got EOF from 127.1.155.130:59431 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 21:46:15 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:15 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:15.411785 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:15.406400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:46751)
0620 21:46:15.406662 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:15.406675 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:15.406773 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:15.407027 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:15.407029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:15.407204 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:15.407310 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:15.407315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:15.408096 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:15.408099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:15.408665 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:15.408673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:15.409141 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:15.409146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:15.409239 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:15.409827 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:15.409883 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:15.411672 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
W20260620 21:46:15.412030 10865 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:15.406480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:46751 (local address 127.1.155.190:46155)
0620 21:46:15.406627 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:15.406631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:15.406749 (+ 118us) server_negotiation.cc:407] Connection header received
0620 21:46:15.406862 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:15.406865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:15.406915 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:15.406994 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:15.407435 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:15.407968 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:15.408779 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:15.408999 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:15.409082 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:15.411867 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:46751: BlockingRecv error: recv got EOF from 127.1.155.131:46751 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 21:46:16 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:16 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:16.047351 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:16.036789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:51175)
0620 21:46:16.037094 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:16.037108 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:16.037202 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:16.037467 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:16.037469 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:16.037642 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:16.037732 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:16.037737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:16.038578 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:16.038581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:16.039181 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:16.039189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:16.039627 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:16.039630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:16.039693 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:16.045069 (+ 5376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:16.045084 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:16.047155 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":92,"threads_started":1}
W20260620 21:46:16.047487 10869 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:16.036943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:51175 (local address 127.1.155.190:46155)
0620 21:46:16.037155 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:16.037158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:16.037171 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:46:16.037292 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:16.037294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:16.037336 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:16.037430 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:16.037873 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:16.038464 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:16.039289 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:16.039524 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:16.039604 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:16.047326 (+ 7722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:51175: BlockingRecv error: recv got EOF from 127.1.155.129:51175 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":58,"threads_started":1}
Jun 20 21:46:16 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:16 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:16.223984 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:16.218675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:57401)
0620 21:46:16.218948 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:16.218969 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:16.219054 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:16.219287 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:16.219289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:16.219460 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:16.219549 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:16.219553 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:16.220277 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:16.220280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:16.220850 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:16.220858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:16.221247 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:16.221250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:16.221306 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:16.221918 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:16.221929 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:16.223865 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":108,"threads_started":1}
W20260620 21:46:16.224156 10869 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:16.218864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:57401 (local address 127.1.155.190:46155)
0620 21:46:16.219005 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:16.219008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:16.219022 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:46:16.219134 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:16.219136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:16.219178 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:16.219248 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:16.219638 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:16.220170 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:16.220943 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:16.221149 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:16.221223 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:16.224000 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:57401: BlockingRecv error: recv got EOF from 127.1.155.130:57401 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 21:46:16 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:16 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:16.418257 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:16.412534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:33255)
0620 21:46:16.412914 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:16.412928 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:16.413066 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:16.413360 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:16.413363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:16.413570 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:16.413654 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:16.413660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:16.414493 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:16.414499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:16.415068 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:16.415076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:16.415569 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:16.415573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:16.415658 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:16.416135 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:16.416146 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:16.418093 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":104,"threads_started":1}
W20260620 21:46:16.418347 10869 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:16.412670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:33255 (local address 127.1.155.190:46155)
0620 21:46:16.412851 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:16.412856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:16.413017 (+ 161us) server_negotiation.cc:407] Connection header received
0620 21:46:16.413176 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:16.413180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:16.413229 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:16.413342 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:16.413774 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:16.414381 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:16.415192 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:16.415455 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:16.415567 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:16.418232 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:33255: BlockingRecv error: recv got EOF from 127.1.155.131:33255 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 21:46:17 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:17 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:17.053660 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:17.048172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:50937)
0620 21:46:17.048439 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:17.048459 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:17.048620 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:17.048852 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:17.048854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:17.049031 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:17.049121 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:17.049126 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:17.050062 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:17.050064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:17.050676 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:17.050685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:17.051165 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:17.051168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:17.051226 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:17.051749 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:17.051760 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:17.053517 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":70,"threads_started":1}
W20260620 21:46:17.053799 10873 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:17.048328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:50937 (local address 127.1.155.190:46155)
0620 21:46:17.048623 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:17.048626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:17.048639 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:46:17.048698 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:17.048701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:17.048740 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:17.048815 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:17.049244 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:17.049967 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:17.050801 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:17.051035 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:17.051155 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:17.053663 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:50937: BlockingRecv error: recv got EOF from 127.1.155.129:50937 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":78,"threads_started":1}
Jun 20 21:46:17 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:17 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:17.230286 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:17.224731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:37957)
0620 21:46:17.225084 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:17.225096 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:17.225218 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:17.225464 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:17.225467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:17.225677 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:17.225762 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:17.225767 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:17.226558 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:17.226562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:17.227169 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:17.227178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:17.227622 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:17.227625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:17.227709 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:17.228190 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:17.228201 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:17.230112 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":107,"threads_started":1}
W20260620 21:46:17.230329 10873 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:17.224843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:37957 (local address 127.1.155.190:46155)
0620 21:46:17.224989 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:17.224994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:17.225169 (+ 175us) server_negotiation.cc:407] Connection header received
0620 21:46:17.225308 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:17.225317 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:17.225359 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:17.225445 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:17.225897 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:17.226425 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:17.227290 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:17.227506 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:17.227621 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:17.230225 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:37957: BlockingRecv error: recv got EOF from 127.1.155.130:37957 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 21:46:17 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:17 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.131@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:17.425149 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:17.419126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.131:45041)
0620 21:46:17.419414 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:17.419427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:17.419525 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:17.419831 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:17.419834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:17.420009 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:17.420101 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:17.420106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:17.420838 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:17.420841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:17.421426 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:17.421434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:17.422042 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:17.422046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:17.422132 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:17.422707 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:17.422725 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:17.424961 (+ 2236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":119,"threads_started":1}
W20260620 21:46:17.425192 10873 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:17.419291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:45041 (local address 127.1.155.190:46155)
0620 21:46:17.419425 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:17.419429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:17.419508 (+ 79us) server_negotiation.cc:407] Connection header received
0620 21:46:17.419599 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:17.419601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:17.419662 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:17.419768 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:17.420212 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:17.420733 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:17.421560 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:17.421922 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:17.422039 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:17.425086 (+ 3047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.131:45041: BlockingRecv error: recv got EOF from 127.1.155.131:45041 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 21:46:18 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:18 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.129@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:18.059731 10876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:18.054494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.129:39183)
0620 21:46:18.054774 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:18.054792 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:18.054889 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:18.055141 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:18.055143 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:18.055316 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:18.055405 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:18.055410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.056161 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.056164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:18.056700 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:18.056707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.057104 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.057106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:18.057172 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:18.057758 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:18.057770 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:18.059617 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":155,"threads_started":1}
W20260620 21:46:18.059966 10877 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:18.054590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:39183 (local address 127.1.155.190:46155)
0620 21:46:18.054809 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:18.054812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:18.054847 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:46:18.054970 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:18.054972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:18.055012 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:18.055088 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:18.055504 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.056050 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.056798 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.057012 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.057087 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:18.059822 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.129:39183: BlockingRecv error: recv got EOF from 127.1.155.129:39183 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
Jun 20 21:46:18 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:46:18 dist-test-slave-5bvn krb5kdc[9818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.155.130@KRBTEST.COM for oryx/127.1.155.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:46:18.236569 10878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:18.231091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46155 (local address 127.1.155.130:57573)
0620 21:46:18.231392 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:18.231409 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:18.231554 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:18.231814 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:18.231817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:18.232031 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:18.232119 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:18.232124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.232881 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.232885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:18.233440 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:18.233447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.233966 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.233970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:18.234055 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:18.234596 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:18.234608 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:18.236387 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.155.190:46155: Server oryx/127.1.155.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":82,"threads_started":1}
W20260620 21:46:18.236608 10877 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:18.231217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:57573 (local address 127.1.155.190:46155)
0620 21:46:18.231383 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:18.231388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:18.231492 (+ 104us) server_negotiation.cc:407] Connection header received
0620 21:46:18.231653 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:18.231656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:18.231709 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:18.231796 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:18.232238 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.232762 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.233559 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.233793 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.233954 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:18.236509 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.155.130:57573: BlockingRecv error: recv got EOF from 127.1.155.130:57573 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 21:46:18.317507 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 10401
I20260620 21:46:18.323129 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 10551
I20260620 21:46:18.328719 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 10686
I20260620 21:46:18.333776 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 10331
2026-06-20T21:46:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18004 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T21:46:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:46:18Z Disabled control of system clock
I20260620 21:46:18.363476 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:34533
--webserver_interface=127.1.155.190
--webserver_port=0
--builtin_ntp_servers=127.1.155.148:37509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:34533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 21:46:18.470186 10882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:18.470425 10882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:18.470459 10882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:18.474300 10882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:46:18.474346 10882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:18.474361 10882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:18.474377 10882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:46:18.474391 10882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:46:18.478981 10882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:37509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:34533
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:34533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:18.479919 10882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:18.481186 10882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:18.489629 10887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:18.489624 10890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:18.489670 10888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:18.490029 10882 server_base.cc:1034] running on GCE node
I20260620 21:46:18.490579 10882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:18.491734 10882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:18.492950 10882 hybrid_clock.cc:648] HybridClock initialized: now 1781991978492925 us; error 49 us; skew 500 ppm
I20260620 21:46:18.494867 10882 webserver.cc:460] Webserver started at http://127.1.155.190:43665/ using document root <none> and password file <none>
I20260620 21:46:18.495443 10882 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:18.495543 10882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:18.495779 10882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:18.497757 10882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "d8279bbc77f94315ab30fc9c01d74e48"
format_stamp: "Formatted at 2026-06-20 21:46:18 on dist-test-slave-5bvn"
server_key: "9dc67624897bbb86a486416ccbc3d857"
server_key_iv: "86793cf32ebffb6242f7a21e604e4b53"
server_key_version: "encryptionkey@0"
I20260620 21:46:18.498381 10882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "d8279bbc77f94315ab30fc9c01d74e48"
format_stamp: "Formatted at 2026-06-20 21:46:18 on dist-test-slave-5bvn"
server_key: "9dc67624897bbb86a486416ccbc3d857"
server_key_iv: "86793cf32ebffb6242f7a21e604e4b53"
server_key_version: "encryptionkey@0"
I20260620 21:46:18.501718 10882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:46:18.504068 10896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:18.505354 10882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:46:18.505481 10882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "d8279bbc77f94315ab30fc9c01d74e48"
format_stamp: "Formatted at 2026-06-20 21:46:18 on dist-test-slave-5bvn"
server_key: "9dc67624897bbb86a486416ccbc3d857"
server_key_iv: "86793cf32ebffb6242f7a21e604e4b53"
server_key_version: "encryptionkey@0"
I20260620 21:46:18.505590 10882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:18.519248 10882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:18.519891 10882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:18.520082 10882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:18.527010 10882 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:34533
I20260620 21:46:18.527040 10948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:34533 every 8 connection(s)
I20260620 21:46:18.527964 10882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:46:18.528787 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 10882
I20260620 21:46:18.528896 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:46:18.529196 1646 external_mini_cluster.cc:1428] Setting key b7ec5c0ea35191ac8eac6b46e1e9f27d
I20260620 21:46:18.531348 10949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:46:18.536895 10949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48: Bootstrap starting.
I20260620 21:46:18.538341 10952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:18.530628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.1.155.190:34533)
0620 21:46:18.531256 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:18.531271 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:18.531304 (+ 33us) server_negotiation.cc:407] Connection header received
0620 21:46:18.531980 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:18.531996 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:18.532359 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:18.532693 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:18.533082 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.534110 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.534973 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.535435 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.535617 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:18.536497 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:18.536501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:18.536515 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:46:18.536918 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:18.537011 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:46:18.537047 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:18.537152 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:18.537309 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 21:46:18.537562 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":148,"threads_started":1}
I20260620 21:46:18.539316 10949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48: Neither blocks nor log segments found. Creating new log.
I20260620 21:46:18.540186 10949 log.cc:824] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48: Log is configured to *not* fsync() on all Append() calls
I20260620 21:46:18.542266 10949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48: No bootstrap required, opened a new log
I20260620 21:46:18.545956 10949 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8279bbc77f94315ab30fc9c01d74e48" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 34533 } }
I20260620 21:46:18.546175 10949 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:46:18.546209 10949 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8279bbc77f94315ab30fc9c01d74e48, State: Initialized, Role: FOLLOWER
I20260620 21:46:18.546618 10949 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8279bbc77f94315ab30fc9c01d74e48" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 34533 } }
I20260620 21:46:18.546736 10949 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:46:18.546805 10949 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:46:18.546905 10949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:46:18.547731 10949 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8279bbc77f94315ab30fc9c01d74e48" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 34533 } }
I20260620 21:46:18.548076 10949 leader_election.cc:302] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8279bbc77f94315ab30fc9c01d74e48; no voters:
I20260620 21:46:18.548461 10949 leader_election.cc:288] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:46:18.548601 10954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:46:18.548852 10954 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [term 1 LEADER]: Becoming Leader. State: Replica: d8279bbc77f94315ab30fc9c01d74e48, State: Running, Role: LEADER
I20260620 21:46:18.549257 10954 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8279bbc77f94315ab30fc9c01d74e48" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 34533 } }
I20260620 21:46:18.549685 10949 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:46:18.551075 10955 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8279bbc77f94315ab30fc9c01d74e48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8279bbc77f94315ab30fc9c01d74e48" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 34533 } } }
I20260620 21:46:18.551211 10955 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [sys.catalog]: This master's current role is: LEADER
I20260620 21:46:18.551440 10956 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8279bbc77f94315ab30fc9c01d74e48. Latest consensus state: current_term: 1 leader_uuid: "d8279bbc77f94315ab30fc9c01d74e48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8279bbc77f94315ab30fc9c01d74e48" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 34533 } } }
I20260620 21:46:18.551599 10956 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48 [sys.catalog]: This master's current role is: LEADER
W20260620 21:46:18.555140 10969 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:46:18.555210 10969 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:46:18.555296 10965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:46:18.556138 10965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:46:18.561275 10965 catalog_manager.cc:1347] Generated new cluster ID: a91108cfb1e54190aa9c37d51689bd24
I20260620 21:46:18.561342 10965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:46:18.571168 10965 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:46:18.572130 10965 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:46:18.584301 10965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8279bbc77f94315ab30fc9c01d74e48: Generated new TSK 0
I20260620 21:46:18.585112 10965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:46:18.591698 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:34533
--builtin_ntp_servers=127.1.155.148:37509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 21:46:18.701700 10973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:18.702015 10973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:18.702091 10973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:18.706291 10973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:18.706343 10973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:18.706426 10973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:46:18.711393 10973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:37509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:34533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:18.712443 10973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:18.713789 10973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:18.723752 10979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:18.723795 10981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:18.723795 10978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:18.723989 10973 server_base.cc:1034] running on GCE node
I20260620 21:46:18.724527 10973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:18.725129 10973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:18.726281 10973 hybrid_clock.cc:648] HybridClock initialized: now 1781991978726263 us; error 32 us; skew 500 ppm
I20260620 21:46:18.728022 10973 webserver.cc:460] Webserver started at http://127.1.155.129:46623/ using document root <none> and password file <none>
I20260620 21:46:18.728586 10973 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:18.728646 10973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:18.728856 10973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:18.730873 10973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "c34f5d113e7948e5a328b06f7ef8a81c"
format_stamp: "Formatted at 2026-06-20 21:46:18 on dist-test-slave-5bvn"
server_key: "23fbd0c9c421075d81f98c5ed23eedda"
server_key_iv: "94f1a1875ade79a0e9c6881f616eec35"
server_key_version: "encryptionkey@0"
I20260620 21:46:18.731410 10973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "c34f5d113e7948e5a328b06f7ef8a81c"
format_stamp: "Formatted at 2026-06-20 21:46:18 on dist-test-slave-5bvn"
server_key: "23fbd0c9c421075d81f98c5ed23eedda"
server_key_iv: "94f1a1875ade79a0e9c6881f616eec35"
server_key_version: "encryptionkey@0"
I20260620 21:46:18.734953 10973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 21:46:18.737404 10987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:18.738670 10973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:46:18.738804 10973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "c34f5d113e7948e5a328b06f7ef8a81c"
format_stamp: "Formatted at 2026-06-20 21:46:18 on dist-test-slave-5bvn"
server_key: "23fbd0c9c421075d81f98c5ed23eedda"
server_key_iv: "94f1a1875ade79a0e9c6881f616eec35"
server_key_version: "encryptionkey@0"
I20260620 21:46:18.738915 10973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:18.754074 10973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:18.754757 10973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:18.754949 10973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:18.755509 10973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:46:18.756423 10973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:46:18.756493 10973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:18.756554 10973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:46:18.756582 10973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:18.765424 10973 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:46529
I20260620 21:46:18.765452 11100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:46529 every 8 connection(s)
I20260620 21:46:18.766317 10973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:46:18.768198 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 10973
I20260620 21:46:18.768316 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:46:18.768620 1646 external_mini_cluster.cc:1428] Setting key 09d1fae3ee0b2d77abd3a674f814c7f0
I20260620 21:46:18.770515 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:34533
--builtin_ntp_servers=127.1.155.148:37509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 21:46:18.775779 10952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:18.768102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:39137 (local address 127.1.155.190:34533)
0620 21:46:18.768244 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:18.768249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:18.769381 (+ 1132us) server_negotiation.cc:407] Connection header received
0620 21:46:18.770841 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:18.770844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:18.770899 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:18.770976 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:18.771953 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.772499 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.773641 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.773912 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.773995 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:18.775249 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:18.775251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:18.775254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:46:18.775265 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:18.775284 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:46:18.775291 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:18.775348 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:18.775592 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 21:46:18.775657 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:46:18.777029 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:18.768548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:34533 (local address 127.1.155.129:39137)
0620 21:46:18.769249 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:18.769299 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:18.770595 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:18.771143 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:18.771150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:18.771230 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:18.771792 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:18.771812 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.772634 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.772639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:18.773533 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:18.773542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.774060 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.774067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:18.774297 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:18.774959 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:46:18.774964 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:18.775017 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:46:18.775105 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:46:18.775375 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:46:18.775386 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:46:18.775395 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 21:46:18.775556 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 21:46:18.775887 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":139,"threads_started":1}
I20260620 21:46:18.778488 11101 heartbeater.cc:347] Connected to a master server at 127.1.155.190:34533
I20260620 21:46:18.778769 11101 heartbeater.cc:464] Registering TS with master...
I20260620 21:46:18.779312 11101 heartbeater.cc:511] Master 127.1.155.190:34533 requested a full tablet report, sending...
I20260620 21:46:18.780814 10913 ts_manager.cc:194] Registered new tserver with Master: c34f5d113e7948e5a328b06f7ef8a81c (127.1.155.129:46529)
I20260620 21:46:18.782032 10913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.155.129:39137
W20260620 21:46:18.882090 11104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:18.882347 11104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:18.882386 11104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:18.886476 11104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:18.886534 11104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:18.886588 11104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:46:18.891300 11104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:37509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:34533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:18.892432 11104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:18.893779 11104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:18.903123 11110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:18.903120 11109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:18.903095 11112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:18.903456 11104 server_base.cc:1034] running on GCE node
I20260620 21:46:18.903874 11104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:18.904623 11104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:18.905814 11104 hybrid_clock.cc:648] HybridClock initialized: now 1781991978905791 us; error 48 us; skew 500 ppm
I20260620 21:46:18.907634 11104 webserver.cc:460] Webserver started at http://127.1.155.130:35903/ using document root <none> and password file <none>
I20260620 21:46:18.908145 11104 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:18.908203 11104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:18.908360 11104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:18.910509 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "018bd612fa9a4b14bc7d98060ddfc56b"
format_stamp: "Formatted at 2026-06-20 21:46:18 on dist-test-slave-5bvn"
server_key: "30c45e63eff1fc19e1443d67de8d757c"
server_key_iv: "a91835c3ff29a8a821ccdf03c75b2bf3"
server_key_version: "encryptionkey@0"
I20260620 21:46:18.911017 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "018bd612fa9a4b14bc7d98060ddfc56b"
format_stamp: "Formatted at 2026-06-20 21:46:18 on dist-test-slave-5bvn"
server_key: "30c45e63eff1fc19e1443d67de8d757c"
server_key_iv: "a91835c3ff29a8a821ccdf03c75b2bf3"
server_key_version: "encryptionkey@0"
I20260620 21:46:18.914455 11104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:46:18.916972 11118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:18.918422 11104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:46:18.918525 11104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "018bd612fa9a4b14bc7d98060ddfc56b"
format_stamp: "Formatted at 2026-06-20 21:46:18 on dist-test-slave-5bvn"
server_key: "30c45e63eff1fc19e1443d67de8d757c"
server_key_iv: "a91835c3ff29a8a821ccdf03c75b2bf3"
server_key_version: "encryptionkey@0"
I20260620 21:46:18.918615 11104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:18.941583 11104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:18.942384 11104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:18.942579 11104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:18.943169 11104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:46:18.944124 11104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:46:18.944170 11104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:18.944234 11104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:46:18.944268 11104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:18.953174 11104 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:33239
I20260620 21:46:18.953198 11231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:33239 every 8 connection(s)
I20260620 21:46:18.954093 11104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:46:18.958279 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 11104
I20260620 21:46:18.958385 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:46:18.958721 1646 external_mini_cluster.cc:1428] Setting key 1aee7449c5dbd633cb6e174df4a75f56
I20260620 21:46:18.960702 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:34533
--builtin_ntp_servers=127.1.155.148:37509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 21:46:18.963843 10952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:18.956142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:49197 (local address 127.1.155.190:34533)
0620 21:46:18.956285 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:18.956289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:18.957331 (+ 1042us) server_negotiation.cc:407] Connection header received
0620 21:46:18.958951 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:18.958955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:18.959015 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:18.959110 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:18.960150 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.960676 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.961911 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.962130 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.962239 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:18.963331 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:18.963334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:18.963337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:46:18.963350 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:18.963376 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:46:18.963384 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:18.963456 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:18.963632 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 21:46:18.963699 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:46:18.964915 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:18.956488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:34533 (local address 127.1.155.130:49197)
0620 21:46:18.957168 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:18.957212 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:18.958702 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:18.959294 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:18.959303 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:18.959410 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:18.960009 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:18.960034 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.960801 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.960804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:18.961747 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:18.961756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:18.962279 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:18.962282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:18.962447 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:18.963058 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:46:18.963060 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:18.963099 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:46:18.963163 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:46:18.963468 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:46:18.963478 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:46:18.963490 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 21:46:18.963621 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 21:46:18.963898 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":122,"threads_started":1}
I20260620 21:46:18.966249 11232 heartbeater.cc:347] Connected to a master server at 127.1.155.190:34533
I20260620 21:46:18.966502 11232 heartbeater.cc:464] Registering TS with master...
I20260620 21:46:18.967031 11232 heartbeater.cc:511] Master 127.1.155.190:34533 requested a full tablet report, sending...
I20260620 21:46:18.968119 10913 ts_manager.cc:194] Registered new tserver with Master: 018bd612fa9a4b14bc7d98060ddfc56b (127.1.155.130:33239)
I20260620 21:46:18.968708 10913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.155.130:49197
W20260620 21:46:19.071991 11235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:19.072257 11235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:19.072333 11235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:19.076437 11235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:19.076501 11235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:19.076556 11235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:46:19.081373 11235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:37509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:34533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:19.082486 11235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:19.083841 11235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:19.093268 11243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:19.093452 11235 server_base.cc:1034] running on GCE node
W20260620 21:46:19.093290 11240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:19.093274 11241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:19.094013 11235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:19.094689 11235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:19.095856 11235 hybrid_clock.cc:648] HybridClock initialized: now 1781991979095828 us; error 44 us; skew 500 ppm
I20260620 21:46:19.097605 11235 webserver.cc:460] Webserver started at http://127.1.155.131:46477/ using document root <none> and password file <none>
I20260620 21:46:19.098266 11235 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:19.098331 11235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:19.098563 11235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:19.100546 11235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "5ad929cacf464d09a90364746208e5a2"
format_stamp: "Formatted at 2026-06-20 21:46:19 on dist-test-slave-5bvn"
server_key: "175fd45e991bf4071a929fcea1465bfc"
server_key_iv: "3b15b850dd89dd68b4b03e1813e825c7"
server_key_version: "encryptionkey@0"
I20260620 21:46:19.101104 11235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "5ad929cacf464d09a90364746208e5a2"
format_stamp: "Formatted at 2026-06-20 21:46:19 on dist-test-slave-5bvn"
server_key: "175fd45e991bf4071a929fcea1465bfc"
server_key_iv: "3b15b850dd89dd68b4b03e1813e825c7"
server_key_version: "encryptionkey@0"
I20260620 21:46:19.104693 11235 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:46:19.107336 11249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:19.108847 11235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:46:19.108966 11235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "5ad929cacf464d09a90364746208e5a2"
format_stamp: "Formatted at 2026-06-20 21:46:19 on dist-test-slave-5bvn"
server_key: "175fd45e991bf4071a929fcea1465bfc"
server_key_iv: "3b15b850dd89dd68b4b03e1813e825c7"
server_key_version: "encryptionkey@0"
I20260620 21:46:19.109081 11235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:19.132885 11235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:19.133608 11235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:19.133803 11235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:19.134442 11235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:46:19.135377 11235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:46:19.135424 11235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:19.135504 11235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:46:19.135543 11235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:19.149996 11235 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:38591
I20260620 21:46:19.150079 11362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:38591 every 8 connection(s)
I20260620 21:46:19.151489 11235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:46:19.157199 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 11235
I20260620 21:46:19.157315 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:46:19.157610 1646 external_mini_cluster.cc:1428] Setting key 3d75fe74b331de2d30b8b5e48b6c71d6
I20260620 21:46:19.163332 10952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:19.153937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:40713 (local address 127.1.155.190:34533)
0620 21:46:19.154098 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:19.154103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:19.155615 (+ 1512us) server_negotiation.cc:407] Connection header received
0620 21:46:19.156809 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:19.156812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:19.156873 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:19.156962 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:19.158277 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:19.159030 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:19.160945 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:19.161273 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:19.161373 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:19.162736 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:19.162740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:19.162742 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:46:19.162777 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:19.162801 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:46:19.162811 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:19.162907 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:19.163096 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 21:46:19.163169 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:46:19.164273 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:19.154513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:34533 (local address 127.1.155.131:40713)
0620 21:46:19.155400 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:19.155470 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:19.156563 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:19.157193 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:19.157202 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:19.157385 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:19.158116 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:19.158133 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:19.159181 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:19.159184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:19.160800 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:19.160811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:19.161459 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:19.161466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:19.161686 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:19.162404 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:46:19.162408 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:19.162477 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:46:19.162557 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:46:19.162915 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:46:19.162926 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:46:19.162933 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 21:46:19.163066 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 21:46:19.163409 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":137,"threads_started":1}
I20260620 21:46:19.166747 11363 heartbeater.cc:347] Connected to a master server at 127.1.155.190:34533
I20260620 21:46:19.166985 11363 heartbeater.cc:464] Registering TS with master...
I20260620 21:46:19.167627 11363 heartbeater.cc:511] Master 127.1.155.190:34533 requested a full tablet report, sending...
I20260620 21:46:19.169202 10913 ts_manager.cc:194] Registered new tserver with Master: 5ad929cacf464d09a90364746208e5a2 (127.1.155.131:38591)
I20260620 21:46:19.169778 10913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.155.131:40713
I20260620 21:46:19.171811 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 21:46:19.174433 10952 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:19.173619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34102 (local address 127.1.155.190:34533)
0620 21:46:19.173737 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:19.173741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:19.174012 (+ 271us) server_negotiation.cc:407] Connection header received
0620 21:46:19.174063 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:19.174065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:19.174102 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:19.174176 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:19.174333 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34102: BlockingRecv error: recv got EOF from 127.0.0.1:34102 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:46:19.174742 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 10973
I20260620 21:46:19.180408 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 11104
I20260620 21:46:19.185632 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 11235
I20260620 21:46:19.190661 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 10882
2026-06-20T21:46:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (855 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T21:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:46:19Z Disabled control of system clock
I20260620 21:46:19.220122 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:38281
--webserver_interface=127.1.155.190
--webserver_port=0
--builtin_ntp_servers=127.1.155.148:36083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:38281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 21:46:19.327636 11374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:19.327901 11374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:19.327937 11374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:19.331923 11374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:46:19.331981 11374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:19.332002 11374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:19.332019 11374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:46:19.332034 11374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:46:19.336670 11374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:36083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:38281
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:38281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:19.338038 11374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:19.339507 11374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:19.348141 11380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:19.348163 11379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:19.348155 11382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:19.348199 11374 server_base.cc:1034] running on GCE node
I20260620 21:46:19.348887 11374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:19.349987 11374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:19.351172 11374 hybrid_clock.cc:648] HybridClock initialized: now 1781991979351153 us; error 29 us; skew 500 ppm
I20260620 21:46:19.352850 11374 webserver.cc:460] Webserver started at http://127.1.155.190:36463/ using document root <none> and password file <none>
I20260620 21:46:19.353410 11374 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:19.353500 11374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:19.353760 11374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:19.355727 11374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "c13e2f13f235478a9b2052ae8046268d"
format_stamp: "Formatted at 2026-06-20 21:46:19 on dist-test-slave-5bvn"
server_key: "2b594dcd7c0dc73baf31cb57e26eedbb"
server_key_iv: "4201e860348e039fd145fc319d27bc3b"
server_key_version: "encryptionkey@0"
I20260620 21:46:19.356280 11374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "c13e2f13f235478a9b2052ae8046268d"
format_stamp: "Formatted at 2026-06-20 21:46:19 on dist-test-slave-5bvn"
server_key: "2b594dcd7c0dc73baf31cb57e26eedbb"
server_key_iv: "4201e860348e039fd145fc319d27bc3b"
server_key_version: "encryptionkey@0"
I20260620 21:46:19.359788 11374 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 21:46:19.362329 11388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:19.363559 11374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:46:19.363708 11374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "c13e2f13f235478a9b2052ae8046268d"
format_stamp: "Formatted at 2026-06-20 21:46:19 on dist-test-slave-5bvn"
server_key: "2b594dcd7c0dc73baf31cb57e26eedbb"
server_key_iv: "4201e860348e039fd145fc319d27bc3b"
server_key_version: "encryptionkey@0"
I20260620 21:46:19.363828 11374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:19.368378 11374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:19.368964 11374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:19.369154 11374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:19.375837 11374 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:38281
I20260620 21:46:19.375859 11440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:38281 every 8 connection(s)
I20260620 21:46:19.376724 11374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:46:19.380125 11441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:46:19.385432 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 11374
I20260620 21:46:19.385533 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:46:19.385797 1646 external_mini_cluster.cc:1428] Setting key 017367e75627ed11851be17dc844c791
I20260620 21:46:19.386260 11441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d: Bootstrap starting.
I20260620 21:46:19.388810 11441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d: Neither blocks nor log segments found. Creating new log.
I20260620 21:46:19.389881 11441 log.cc:824] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:46:19.391580 11444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:19.387291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.1.155.190:38281)
0620 21:46:19.387906 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:19.387921 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:19.387955 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:46:19.388570 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:19.388586 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:19.388977 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:19.389254 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:19.390158 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:19.390162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:19.390173 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:46:19.390184 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:19.390237 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:46:19.390253 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:19.390376 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:19.390482 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 21:46:19.390714 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":141,"threads_started":1}
I20260620 21:46:19.392181 11441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d: No bootstrap required, opened a new log
I20260620 21:46:19.395736 11441 raft_consensus.cc:348] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13e2f13f235478a9b2052ae8046268d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38281 } }
I20260620 21:46:19.395941 11441 raft_consensus.cc:374] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:46:19.395982 11441 raft_consensus.cc:729] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13e2f13f235478a9b2052ae8046268d, State: Initialized, Role: FOLLOWER
I20260620 21:46:19.396313 11441 consensus_queue.cc:260] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13e2f13f235478a9b2052ae8046268d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38281 } }
I20260620 21:46:19.396417 11441 raft_consensus.cc:388] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:46:19.396454 11441 raft_consensus.cc:482] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:46:19.396529 11441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:46:19.397351 11441 raft_consensus.cc:504] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13e2f13f235478a9b2052ae8046268d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38281 } }
I20260620 21:46:19.397637 11441 leader_election.cc:302] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c13e2f13f235478a9b2052ae8046268d; no voters:
I20260620 21:46:19.397929 11441 leader_election.cc:288] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:46:19.398020 11446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:46:19.398276 11446 raft_consensus.cc:686] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [term 1 LEADER]: Becoming Leader. State: Replica: c13e2f13f235478a9b2052ae8046268d, State: Running, Role: LEADER
I20260620 21:46:19.398597 11446 consensus_queue.cc:237] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13e2f13f235478a9b2052ae8046268d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38281 } }
I20260620 21:46:19.399174 11441 sys_catalog.cc:564] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:46:19.400247 11448 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c13e2f13f235478a9b2052ae8046268d. Latest consensus state: current_term: 1 leader_uuid: "c13e2f13f235478a9b2052ae8046268d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13e2f13f235478a9b2052ae8046268d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38281 } } }
I20260620 21:46:19.400388 11448 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [sys.catalog]: This master's current role is: LEADER
I20260620 21:46:19.400712 11447 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c13e2f13f235478a9b2052ae8046268d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13e2f13f235478a9b2052ae8046268d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38281 } } }
I20260620 21:46:19.400797 11447 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d [sys.catalog]: This master's current role is: LEADER
I20260620 21:46:19.400897 11455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:46:19.403561 11455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:46:19.409288 11455 catalog_manager.cc:1347] Generated new cluster ID: 1ee759d9a8d84de1969a285663dbeb65
I20260620 21:46:19.409358 11455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:46:19.418118 11455 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:46:19.418406 11455 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:46:19.428767 11455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c13e2f13f235478a9b2052ae8046268d: Generated new TSK 0
I20260620 21:46:19.429446 11455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:46:19.438302 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:38281
--builtin_ntp_servers=127.1.155.148:36083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 21:46:19.546592 11465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:19.546841 11465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:19.546926 11465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:19.551000 11465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:19.551054 11465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:19.551156 11465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:46:19.556108 11465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:36083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:19.557205 11465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:19.558645 11465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:19.568089 11473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:19.568259 11465 server_base.cc:1034] running on GCE node
W20260620 21:46:19.568089 11470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:19.568106 11471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:19.568783 11465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:19.569420 11465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:19.570613 11465 hybrid_clock.cc:648] HybridClock initialized: now 1781991979570576 us; error 48 us; skew 500 ppm
I20260620 21:46:19.572361 11465 webserver.cc:460] Webserver started at http://127.1.155.129:41813/ using document root <none> and password file <none>
I20260620 21:46:19.572942 11465 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:19.573038 11465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:19.573261 11465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:19.575314 11465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "ba65bfee2d45496496bfc0096810690a"
format_stamp: "Formatted at 2026-06-20 21:46:19 on dist-test-slave-5bvn"
server_key: "99a75ba25c9929f1f723b9cef7671ad6"
server_key_iv: "e1188a60404432a0a5701f46ea1f7181"
server_key_version: "encryptionkey@0"
I20260620 21:46:19.575888 11465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "ba65bfee2d45496496bfc0096810690a"
format_stamp: "Formatted at 2026-06-20 21:46:19 on dist-test-slave-5bvn"
server_key: "99a75ba25c9929f1f723b9cef7671ad6"
server_key_iv: "e1188a60404432a0a5701f46ea1f7181"
server_key_version: "encryptionkey@0"
I20260620 21:46:19.579521 11465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:46:19.582368 11479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:19.583639 11465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:46:19.583796 11465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "ba65bfee2d45496496bfc0096810690a"
format_stamp: "Formatted at 2026-06-20 21:46:19 on dist-test-slave-5bvn"
server_key: "99a75ba25c9929f1f723b9cef7671ad6"
server_key_iv: "e1188a60404432a0a5701f46ea1f7181"
server_key_version: "encryptionkey@0"
I20260620 21:46:19.583927 11465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:19.588445 11465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:19.589044 11465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:19.589224 11465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:19.589763 11465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:46:19.590816 11465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:46:19.590890 11465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:19.590991 11465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:46:19.591041 11465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:19.600040 11465 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:36447
I20260620 21:46:19.600059 11592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:36447 every 8 connection(s)
I20260620 21:46:19.600998 11465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:46:19.604436 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 11465
I20260620 21:46:19.604557 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:46:19.604866 1646 external_mini_cluster.cc:1428] Setting key b38d718876b303dbdd0993e4dd4d30fc
I20260620 21:46:19.607004 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:38281
--builtin_ntp_servers=127.1.155.148:36083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 21:46:19.607963 11444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:19.603030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:52015 (local address 127.1.155.190:38281)
0620 21:46:19.603174 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:19.603178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:19.604357 (+ 1179us) server_negotiation.cc:407] Connection header received
0620 21:46:19.606007 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:19.606010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:19.606051 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:19.606155 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:19.607402 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:19.607409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:19.607415 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:46:19.607432 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:19.607460 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:46:19.607485 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:19.607551 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:19.607788 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 21:46:19.607830 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:46:19.608958 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:19.603445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:38281 (local address 127.1.155.129:52015)
0620 21:46:19.604181 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:19.604238 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:19.605740 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:19.606348 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:19.606356 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:19.606468 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:19.607077 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:46:19.607079 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:19.607140 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:46:19.607237 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:46:19.607624 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:46:19.607635 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:46:19.607644 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 21:46:19.607811 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 21:46:19.608052 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
I20260620 21:46:19.610261 11593 heartbeater.cc:347] Connected to a master server at 127.1.155.190:38281
I20260620 21:46:19.610512 11593 heartbeater.cc:464] Registering TS with master...
I20260620 21:46:19.610970 11593 heartbeater.cc:511] Master 127.1.155.190:38281 requested a full tablet report, sending...
I20260620 21:46:19.612708 11405 ts_manager.cc:194] Registered new tserver with Master: ba65bfee2d45496496bfc0096810690a (127.1.155.129:36447)
W20260620 21:46:19.717806 11596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:19.718111 11596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:19.718151 11596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:19.722193 11596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:19.722250 11596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:19.722308 11596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:46:19.726964 11596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:36083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:19.727983 11596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:19.729269 11596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:19.738638 11604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:19.738638 11601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:19.738730 11602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:19.739147 11596 server_base.cc:1034] running on GCE node
I20260620 21:46:19.739730 11596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:19.740477 11596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:19.741670 11596 hybrid_clock.cc:648] HybridClock initialized: now 1781991979741648 us; error 34 us; skew 500 ppm
I20260620 21:46:19.743892 11596 webserver.cc:460] Webserver started at http://127.1.155.130:43685/ using document root <none> and password file <none>
I20260620 21:46:19.744524 11596 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:19.744599 11596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:19.744853 11596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:19.747169 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "b8e521af34d249a2ad1badf4ff1ea594"
format_stamp: "Formatted at 2026-06-20 21:46:19 on dist-test-slave-5bvn"
server_key: "417bc3afcb3eba7ed550735a2aae07ff"
server_key_iv: "1b74a0ade45fbcc714d15510ee358790"
server_key_version: "encryptionkey@0"
I20260620 21:46:19.747823 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "b8e521af34d249a2ad1badf4ff1ea594"
format_stamp: "Formatted at 2026-06-20 21:46:19 on dist-test-slave-5bvn"
server_key: "417bc3afcb3eba7ed550735a2aae07ff"
server_key_iv: "1b74a0ade45fbcc714d15510ee358790"
server_key_version: "encryptionkey@0"
I20260620 21:46:19.752039 11596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:46:19.754761 11610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:19.756106 11596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:46:19.756274 11596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "b8e521af34d249a2ad1badf4ff1ea594"
format_stamp: "Formatted at 2026-06-20 21:46:19 on dist-test-slave-5bvn"
server_key: "417bc3afcb3eba7ed550735a2aae07ff"
server_key_iv: "1b74a0ade45fbcc714d15510ee358790"
server_key_version: "encryptionkey@0"
I20260620 21:46:19.756449 11596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:19.761560 11596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:19.762264 11596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:19.762476 11596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:19.763056 11596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:46:19.764017 11596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:46:19.764089 11596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:19.764161 11596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:46:19.764214 11596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:19.773840 11596 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:39489
I20260620 21:46:19.773895 11723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:39489 every 8 connection(s)
I20260620 21:46:19.774813 11596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:46:19.781116 11444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:19.776595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:45271 (local address 127.1.155.190:38281)
0620 21:46:19.776743 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:19.776748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:19.777736 (+ 988us) server_negotiation.cc:407] Connection header received
0620 21:46:19.779267 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:19.779271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:19.779322 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:19.779442 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:19.780622 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:19.780625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:19.780628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:46:19.780647 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:19.780667 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:46:19.780675 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:19.780778 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:19.780924 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 21:46:19.780973 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:46:19.782143 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:19.776899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:38281 (local address 127.1.155.130:45271)
0620 21:46:19.777571 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:19.777614 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:19.779031 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:19.779627 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:19.779634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:19.779726 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:19.780345 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:46:19.780346 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:19.780387 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:46:19.780459 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:46:19.780760 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:46:19.780768 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:46:19.780773 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:46:19.780980 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 21:46:19.781227 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":107,"threads_started":1}
I20260620 21:46:19.783342 11724 heartbeater.cc:347] Connected to a master server at 127.1.155.190:38281
I20260620 21:46:19.783586 11724 heartbeater.cc:464] Registering TS with master...
I20260620 21:46:19.783900 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 11596
I20260620 21:46:19.784022 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:46:19.784042 11724 heartbeater.cc:511] Master 127.1.155.190:38281 requested a full tablet report, sending...
I20260620 21:46:19.784303 1646 external_mini_cluster.cc:1428] Setting key 6b51e985e1149054ff7a597000842dd5
I20260620 21:46:19.785142 11405 ts_manager.cc:194] Registered new tserver with Master: b8e521af34d249a2ad1badf4ff1ea594 (127.1.155.130:39489)
I20260620 21:46:19.786388 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:38281
--builtin_ntp_servers=127.1.155.148:36083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 21:46:19.901473 11727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:19.901790 11727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:19.901885 11727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:19.906145 11727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:19.906200 11727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:19.906289 11727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:46:19.911281 11727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:36083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:19.912389 11727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:19.913743 11727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:19.923113 11733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:19.923134 11735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:19.923128 11732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:19.923378 11727 server_base.cc:1034] running on GCE node
I20260620 21:46:19.924002 11727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:19.924683 11727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:19.925899 11727 hybrid_clock.cc:648] HybridClock initialized: now 1781991979925874 us; error 56 us; skew 500 ppm
I20260620 21:46:19.927750 11727 webserver.cc:460] Webserver started at http://127.1.155.131:36239/ using document root <none> and password file <none>
I20260620 21:46:19.928413 11727 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:19.928510 11727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:19.928727 11727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:19.930848 11727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "013f2460c7e2499d9ca0a7b043e211d7"
format_stamp: "Formatted at 2026-06-20 21:46:19 on dist-test-slave-5bvn"
server_key: "8a880605694fde47cd62e7d179a04fa0"
server_key_iv: "0a4077b30bdc93cda3c47d3e490f8d6e"
server_key_version: "encryptionkey@0"
I20260620 21:46:19.931416 11727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "013f2460c7e2499d9ca0a7b043e211d7"
format_stamp: "Formatted at 2026-06-20 21:46:19 on dist-test-slave-5bvn"
server_key: "8a880605694fde47cd62e7d179a04fa0"
server_key_iv: "0a4077b30bdc93cda3c47d3e490f8d6e"
server_key_version: "encryptionkey@0"
I20260620 21:46:19.935206 11727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:46:19.937682 11741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:19.939080 11727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:46:19.939229 11727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "013f2460c7e2499d9ca0a7b043e211d7"
format_stamp: "Formatted at 2026-06-20 21:46:19 on dist-test-slave-5bvn"
server_key: "8a880605694fde47cd62e7d179a04fa0"
server_key_iv: "0a4077b30bdc93cda3c47d3e490f8d6e"
server_key_version: "encryptionkey@0"
I20260620 21:46:19.939358 11727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:19.943898 11727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:19.944496 11727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:19.944684 11727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:19.945272 11727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:46:19.946230 11727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:46:19.946306 11727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:19.946393 11727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:46:19.946444 11727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:19.955118 11727 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:36301
I20260620 21:46:19.955163 11854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:36301 every 8 connection(s)
I20260620 21:46:19.956061 11727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:46:19.962108 11444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:19.957815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:42339 (local address 127.1.155.190:38281)
0620 21:46:19.958034 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:19.958038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:19.959046 (+ 1008us) server_negotiation.cc:407] Connection header received
0620 21:46:19.960414 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:19.960418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:19.960460 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:19.960542 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:19.961608 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:19.961612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:19.961615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:46:19.961626 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:19.961653 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:46:19.961662 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:19.961760 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:19.961927 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 21:46:19.961971 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 21:46:19.962603 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 11727
I20260620 21:46:19.962728 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:46:19.962988 1646 external_mini_cluster.cc:1428] Setting key a0a22c2f4365f46de748cdfb538a658a
I20260620 21:46:19.963079 11857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:19.958253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:38281 (local address 127.1.155.131:42339)
0620 21:46:19.958901 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:19.958940 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:19.960210 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:19.960699 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:19.960706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:19.960790 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:19.961354 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:46:19.961355 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:19.961394 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:46:19.961460 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:46:19.961745 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:46:19.961753 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:46:19.961758 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:46:19.961966 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 21:46:19.962197 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":110,"threads_started":1}
I20260620 21:46:19.964299 11855 heartbeater.cc:347] Connected to a master server at 127.1.155.190:38281
I20260620 21:46:19.964521 11855 heartbeater.cc:464] Registering TS with master...
I20260620 21:46:19.964975 11855 heartbeater.cc:511] Master 127.1.155.190:38281 requested a full tablet report, sending...
I20260620 21:46:19.966117 11405 ts_manager.cc:194] Registered new tserver with Master: 013f2460c7e2499d9ca0a7b043e211d7 (127.1.155.131:36301)
I20260620 21:46:19.977582 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 21:46:19.980805 11444 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:46:19.979849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.1.155.190:38281)
0620 21:46:19.980033 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:19.980038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:19.980192 (+ 154us) server_negotiation.cc:407] Connection header received
0620 21:46:19.980269 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:19.980271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:19.980307 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:19.980382 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:19.980637 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52700: BlockingRecv error: recv got EOF from 127.0.0.1:52700 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 21:46:19.981017 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 11465
I20260620 21:46:19.986852 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 11596
I20260620 21:46:19.992643 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 11727
I20260620 21:46:19.998272 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 11374
2026-06-20T21:46:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (808 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:46:20 dist-test-slave-5bvn krb5kdc[11865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:46:20 dist-test-slave-5bvn krb5kdc[11865](info): set up 2 sockets
Jun 20 21:46:20 dist-test-slave-5bvn krb5kdc[11865](info): commencing operation
krb5kdc: starting...
W20260620 21:46:21.050902 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:46:21 dist-test-slave-5bvn krb5kdc[11865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:46:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:46:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:46:21.181214 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:45907
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:45907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:46:21.291850 11878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:21.292102 11878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:21.292143 11878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:21.296113 11878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:46:21.296196 11878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:21.296217 11878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:21.296234 11878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:46:21.296296 11878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:46:21.301556 11878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:45907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:45907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:21.302632 11878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:21.303633 11878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:21.312114 11883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:21.312191 11886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:21.312088 11884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:21.312481 11878 server_base.cc:1034] running on GCE node
I20260620 21:46:21.313006 11878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:21.314142 11878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:21.315299 11878 hybrid_clock.cc:648] HybridClock initialized: now 1781991981315274 us; error 38 us; skew 500 ppm
Jun 20 21:46:21 dist-test-slave-5bvn krb5kdc[11865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:46:21.322258 11878 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:46:21.323098 11878 webserver.cc:460] Webserver started at http://127.1.155.190:39755/ using document root <none> and password file <none>
I20260620 21:46:21.323616 11878 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:21.323673 11878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:21.323843 11878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:21.325752 11878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "e5f0b5319e5d4e11ac60e782ee58ce31"
format_stamp: "Formatted at 2026-06-20 21:46:21 on dist-test-slave-5bvn"
server_key: "a3f75fe7e6d872797b93755576d9f0a1"
server_key_iv: "013d12dea483943f787f579a84192730"
server_key_version: "encryptionkey@0"
I20260620 21:46:21.326336 11878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "e5f0b5319e5d4e11ac60e782ee58ce31"
format_stamp: "Formatted at 2026-06-20 21:46:21 on dist-test-slave-5bvn"
server_key: "a3f75fe7e6d872797b93755576d9f0a1"
server_key_iv: "013d12dea483943f787f579a84192730"
server_key_version: "encryptionkey@0"
I20260620 21:46:21.329768 11878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:46:21.332056 11893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:21.333346 11878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:46:21.333451 11878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "e5f0b5319e5d4e11ac60e782ee58ce31"
format_stamp: "Formatted at 2026-06-20 21:46:21 on dist-test-slave-5bvn"
server_key: "a3f75fe7e6d872797b93755576d9f0a1"
server_key_iv: "013d12dea483943f787f579a84192730"
server_key_version: "encryptionkey@0"
I20260620 21:46:21.333595 11878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:21.358335 11878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:21.361864 11878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:21.362071 11878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:21.368849 11878 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:45907
I20260620 21:46:21.369140 11945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:45907 every 8 connection(s)
I20260620 21:46:21.369812 11878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:46:21.372442 11946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:46:21.377991 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 11878
I20260620 21:46:21.378116 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:46:21.378412 1646 external_mini_cluster.cc:1428] Setting key 89dd75cdccf2585351b95f7f5cf3da8b
I20260620 21:46:21.378400 11946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31: Bootstrap starting.
I20260620 21:46:21.380795 11946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31: Neither blocks nor log segments found. Creating new log.
I20260620 21:46:21.381608 11946 log.cc:824] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31: Log is configured to *not* fsync() on all Append() calls
I20260620 21:46:21.383630 11946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31: No bootstrap required, opened a new log
Jun 20 21:46:21 dist-test-slave-5bvn krb5kdc[11865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:46:21.387897 11946 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f0b5319e5d4e11ac60e782ee58ce31" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45907 } }
I20260620 21:46:21.388128 11946 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:46:21.388207 11946 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f0b5319e5d4e11ac60e782ee58ce31, State: Initialized, Role: FOLLOWER
I20260620 21:46:21.388674 11946 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f0b5319e5d4e11ac60e782ee58ce31" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45907 } }
I20260620 21:46:21.388831 11946 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:46:21.388906 11946 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:46:21.389016 11946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:46:21.390146 11946 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f0b5319e5d4e11ac60e782ee58ce31" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45907 } }
I20260620 21:46:21.390491 11946 leader_election.cc:302] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5f0b5319e5d4e11ac60e782ee58ce31; no voters:
I20260620 21:46:21.390801 11946 leader_election.cc:288] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:46:21.390894 11951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:46:21.391126 11951 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [term 1 LEADER]: Becoming Leader. State: Replica: e5f0b5319e5d4e11ac60e782ee58ce31, State: Running, Role: LEADER
I20260620 21:46:21.391453 11951 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f0b5319e5d4e11ac60e782ee58ce31" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45907 } }
I20260620 21:46:21.392189 11946 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:46:21.393179 11953 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5f0b5319e5d4e11ac60e782ee58ce31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f0b5319e5d4e11ac60e782ee58ce31" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45907 } } }
I20260620 21:46:21.393325 11953 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [sys.catalog]: This master's current role is: LEADER
I20260620 21:46:21.393576 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:21.379640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.1.155.190:45907)
0620 21:46:21.380022 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:21.380047 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:21.380082 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:46:21.380701 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:21.380718 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:21.381092 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:21.381363 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:21.382268 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:21.383325 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:21.384470 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:21.384936 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:21.385055 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:21.387786 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:21.387792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:21.387805 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:46:21.387837 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:21.389909 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:21.390333 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:21.390345 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:21.390350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:21.390476 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:21.390767 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:21.390772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:21.390774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:21.391324 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:21.392102 (+ 778us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:21.392266 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 21:46:21.392567 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":137,"threads_started":1}
I20260620 21:46:21.393713 11960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:46:21.393599 11952 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f0b5319e5d4e11ac60e782ee58ce31. Latest consensus state: current_term: 1 leader_uuid: "e5f0b5319e5d4e11ac60e782ee58ce31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f0b5319e5d4e11ac60e782ee58ce31" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45907 } } }
I20260620 21:46:21.394003 11952 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31 [sys.catalog]: This master's current role is: LEADER
I20260620 21:46:21.396808 11960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:46:21.402382 11960 catalog_manager.cc:1347] Generated new cluster ID: 2f884d3adeb247229ae742289cc2a6e2
I20260620 21:46:21.402451 11960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:46:21.425150 11960 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:46:21.426193 11960 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:46:21.436182 11960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31: Generated new TSK 0
I20260620 21:46:21.436955 11960 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:46:21.470906 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:45907
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:46:21.604089 11972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:21.604385 11972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:21.604463 11972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:21.608521 11972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:21.608613 11972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:21.608711 11972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:46:21.614281 11972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:45907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:21.615335 11972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:21.616343 11972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:21.625764 11980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:21.625784 11978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:21.625823 11972 server_base.cc:1034] running on GCE node
W20260620 21:46:21.625761 11977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:21.626437 11972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:21.627190 11972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:21.628369 11972 hybrid_clock.cc:648] HybridClock initialized: now 1781991981628357 us; error 26 us; skew 500 ppm
Jun 20 21:46:21 dist-test-slave-5bvn krb5kdc[11865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:46:21.631242 11972 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:46:21.632056 11972 webserver.cc:460] Webserver started at http://127.1.155.129:33795/ using document root <none> and password file <none>
I20260620 21:46:21.632558 11972 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:21.632613 11972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:21.632773 11972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:21.634821 11972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "da48b3be20954949b190bedfee030395"
format_stamp: "Formatted at 2026-06-20 21:46:21 on dist-test-slave-5bvn"
server_key: "a60c3e1ad85155a8b2602f3eefad80b1"
server_key_iv: "58c0de7aaa920a187782c58bcbe79dc2"
server_key_version: "encryptionkey@0"
I20260620 21:46:21.635326 11972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "da48b3be20954949b190bedfee030395"
format_stamp: "Formatted at 2026-06-20 21:46:21 on dist-test-slave-5bvn"
server_key: "a60c3e1ad85155a8b2602f3eefad80b1"
server_key_iv: "58c0de7aaa920a187782c58bcbe79dc2"
server_key_version: "encryptionkey@0"
I20260620 21:46:21.638993 11972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:46:21.641351 11987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:21.642683 11972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:46:21.642817 11972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "da48b3be20954949b190bedfee030395"
format_stamp: "Formatted at 2026-06-20 21:46:21 on dist-test-slave-5bvn"
server_key: "a60c3e1ad85155a8b2602f3eefad80b1"
server_key_iv: "58c0de7aaa920a187782c58bcbe79dc2"
server_key_version: "encryptionkey@0"
I20260620 21:46:21.642928 11972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:21.658366 11972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:21.661634 11972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:21.661906 11972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:21.662508 11972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:46:21.663331 11972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:46:21.663378 11972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:21.663419 11972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:46:21.663470 11972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:21.672329 11972 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:35047
I20260620 21:46:21.672349 12100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:35047 every 8 connection(s)
I20260620 21:46:21.673202 11972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:46:21.678086 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 11972
I20260620 21:46:21.678217 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:46:21.678488 1646 external_mini_cluster.cc:1428] Setting key 8c261430f27b7f82984a0514c587aa9b
Jun 20 21:46:21 dist-test-slave-5bvn krb5kdc[11865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
I20260620 21:46:21.687661 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:21.675032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:53857 (local address 127.1.155.190:45907)
0620 21:46:21.675176 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:21.675181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:21.676002 (+ 821us) server_negotiation.cc:407] Connection header received
0620 21:46:21.677383 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:21.677386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:21.677431 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:21.677516 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:21.678929 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:21.679459 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:21.680596 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:21.680833 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:21.680943 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:21.684046 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:21.684049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:21.684051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:46:21.684074 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:21.685506 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:21.686109 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:21.686113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:21.686115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:21.686196 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:21.686555 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:21.686558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:21.686560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:21.686745 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:21.686861 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:21.687368 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 21:46:21.687495 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:46:21.688541 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:21.675380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:45907 (local address 127.1.155.129:53857)
0620 21:46:21.675840 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:21.675893 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:21.677174 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:21.677693 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:21.677699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:21.678154 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:21.678758 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:21.678774 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:21.679587 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:21.679592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:21.680461 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:21.680470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:21.680981 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:21.680985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:21.681151 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:21.681867 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:21.681892 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:21.683864 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:46:21.685632 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:46:21.685639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:46:21.685656 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:46:21.686005 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:46:21.686301 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:46:21.686304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:46:21.686306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:46:21.686450 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:46:21.686859 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:46:21.686866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:46:21.687119 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 21:46:21.687308 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 21:46:21.687666 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":143,"threads_started":1}
I20260620 21:46:21.689932 12101 heartbeater.cc:347] Connected to a master server at 127.1.155.190:45907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
I20260620 21:46:21.690199 12101 heartbeater.cc:464] Registering TS with master...
I20260620 21:46:21.690750 12101 heartbeater.cc:511] Master 127.1.155.190:45907 requested a full tablet report, sending...
I20260620 21:46:21.692302 11910 ts_manager.cc:194] Registered new tserver with Master: da48b3be20954949b190bedfee030395 (127.1.155.129:35047)
I20260620 21:46:21.694044 11910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:53857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:46:21.707093 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:45907
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:46:21.816200 12106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:21.816440 12106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:21.816483 12106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:21.820964 12106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:21.821030 12106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:21.821118 12106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:46:21.829947 12106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:45907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:21.831401 12106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:21.832839 12106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:21.845077 12112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:21.845137 12111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:21.845423 12114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:21.845791 12106 server_base.cc:1034] running on GCE node
I20260620 21:46:21.846282 12106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:21.847098 12106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:21.848289 12106 hybrid_clock.cc:648] HybridClock initialized: now 1781991981848261 us; error 66 us; skew 500 ppm
Jun 20 21:46:21 dist-test-slave-5bvn krb5kdc[11865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:46:21.851511 12106 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:46:21.852411 12106 webserver.cc:460] Webserver started at http://127.1.155.130:33913/ using document root <none> and password file <none>
I20260620 21:46:21.852993 12106 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:21.853048 12106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:21.853281 12106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:21.855376 12106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "97a3e1a635304d2aa0a8432f7050e10d"
format_stamp: "Formatted at 2026-06-20 21:46:21 on dist-test-slave-5bvn"
server_key: "db9430213e312e4785256a4c6d147f76"
server_key_iv: "10e35a6bc2681cec77a86f9bca2cc452"
server_key_version: "encryptionkey@0"
I20260620 21:46:21.855935 12106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "97a3e1a635304d2aa0a8432f7050e10d"
format_stamp: "Formatted at 2026-06-20 21:46:21 on dist-test-slave-5bvn"
server_key: "db9430213e312e4785256a4c6d147f76"
server_key_iv: "10e35a6bc2681cec77a86f9bca2cc452"
server_key_version: "encryptionkey@0"
I20260620 21:46:21.859965 12106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:46:21.862500 12121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:21.864087 12106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:46:21.864216 12106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "97a3e1a635304d2aa0a8432f7050e10d"
format_stamp: "Formatted at 2026-06-20 21:46:21 on dist-test-slave-5bvn"
server_key: "db9430213e312e4785256a4c6d147f76"
server_key_iv: "10e35a6bc2681cec77a86f9bca2cc452"
server_key_version: "encryptionkey@0"
I20260620 21:46:21.864331 12106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:21.879146 12106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:21.881929 12106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:21.882117 12106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:21.882670 12106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:46:21.883524 12106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:46:21.883602 12106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:21.883702 12106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:46:21.883749 12106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:21.892081 12106 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:35277
I20260620 21:46:21.892093 12234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:35277 every 8 connection(s)
I20260620 21:46:21.893231 12106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
Jun 20 21:46:21 dist-test-slave-5bvn krb5kdc[11865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:46:21.903473 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 12106
I20260620 21:46:21.903605 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:46:21.903957 1646 external_mini_cluster.cc:1428] Setting key f1be1a0b141b046daf0f4066473e555c
I20260620 21:46:21.908058 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:21.895287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:42759 (local address 127.1.155.190:45907)
0620 21:46:21.895438 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:21.895442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:21.896312 (+ 870us) server_negotiation.cc:407] Connection header received
0620 21:46:21.897701 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:21.897704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:21.897768 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:21.897924 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:21.899069 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:21.899599 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:21.900713 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:21.900939 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:21.901054 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:21.903540 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:21.903543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:21.903545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:46:21.903574 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:21.905814 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:21.906316 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:21.906323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:21.906327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:21.906422 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:21.906740 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:21.906746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:21.906751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:21.907000 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:21.907111 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:21.907837 (+ 726us) server_negotiation.cc:299] Negotiation successful
0620 21:46:21.907941 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:46:21.909080 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:21.895704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:45907 (local address 127.1.155.130:42759)
0620 21:46:21.896162 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:21.896205 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:21.897478 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:21.898091 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:21.898099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:21.898456 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:21.898924 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:21.898937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:21.899725 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:21.899729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:21.900578 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:21.900588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:21.901088 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:21.901092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:21.901262 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:21.901918 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:21.901935 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:21.903367 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:46:21.905953 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:46:21.905958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:46:21.905969 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:46:21.906213 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:46:21.906516 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:46:21.906518 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:46:21.906519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:46:21.906649 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:46:21.907168 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:46:21.907178 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:46:21.907538 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 21:46:21.907786 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 21:46:21.908165 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":124,"threads_started":1}
I20260620 21:46:21.910331 12235 heartbeater.cc:347] Connected to a master server at 127.1.155.190:45907
I20260620 21:46:21.910596 12235 heartbeater.cc:464] Registering TS with master...
I20260620 21:46:21.911234 12235 heartbeater.cc:511] Master 127.1.155.190:45907 requested a full tablet report, sending...
I20260620 21:46:21.912475 11910 ts_manager.cc:194] Registered new tserver with Master: 97a3e1a635304d2aa0a8432f7050e10d (127.1.155.130:35277)
I20260620 21:46:21.913329 11910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:42759
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:46:21.933005 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:45907
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:46:22.043285 12240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:46:22.043516 12240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:46:22.043560 12240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:46:22.047751 12240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:46:22.047813 12240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:46:22.047899 12240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:46:22.053825 12240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:45907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:46:22.054940 12240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:46:22.055929 12240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:46:22.065642 12245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:22.065742 12240 server_base.cc:1034] running on GCE node
W20260620 21:46:22.065732 12246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:46:22.065642 12248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:46:22.066349 12240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:46:22.067070 12240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:46:22.068266 12240 hybrid_clock.cc:648] HybridClock initialized: now 1781991982068247 us; error 33 us; skew 500 ppm
Jun 20 21:46:22 dist-test-slave-5bvn krb5kdc[11865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991982, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:46:22.071218 12240 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:46:22.072105 12240 webserver.cc:460] Webserver started at http://127.1.155.131:42179/ using document root <none> and password file <none>
I20260620 21:46:22.072669 12240 fs_manager.cc:359] Metadata directory not provided
I20260620 21:46:22.072759 12240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:46:22.072983 12240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:46:22.074999 12240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "9b151a4dbd464f9aa0f933126f0bd095"
format_stamp: "Formatted at 2026-06-20 21:46:22 on dist-test-slave-5bvn"
server_key: "6c2ef1c995ab77068a50945ccf9cb859"
server_key_iv: "3394ac3b6c3a67fc13692502e5cf8b4f"
server_key_version: "encryptionkey@0"
I20260620 21:46:22.075536 12240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "9b151a4dbd464f9aa0f933126f0bd095"
format_stamp: "Formatted at 2026-06-20 21:46:22 on dist-test-slave-5bvn"
server_key: "6c2ef1c995ab77068a50945ccf9cb859"
server_key_iv: "3394ac3b6c3a67fc13692502e5cf8b4f"
server_key_version: "encryptionkey@0"
I20260620 21:46:22.079180 12240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:46:22.081633 12255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:22.082968 12240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:46:22.083117 12240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "9b151a4dbd464f9aa0f933126f0bd095"
format_stamp: "Formatted at 2026-06-20 21:46:22 on dist-test-slave-5bvn"
server_key: "6c2ef1c995ab77068a50945ccf9cb859"
server_key_iv: "3394ac3b6c3a67fc13692502e5cf8b4f"
server_key_version: "encryptionkey@0"
I20260620 21:46:22.083240 12240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:46:22.098495 12240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:46:22.101785 12240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:46:22.102047 12240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:46:22.102654 12240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:46:22.103500 12240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:46:22.103575 12240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:22.103644 12240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:46:22.103694 12240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:46:22.112440 12240 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:34473
I20260620 21:46:22.112465 12368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:34473 every 8 connection(s)
I20260620 21:46:22.113451 12240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:46:22.119405 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 12240
I20260620 21:46:22.119549 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:46:22.119879 1646 external_mini_cluster.cc:1428] Setting key 4604dbe3bf815d2ca07abe76e5b69273
Jun 20 21:46:22 dist-test-slave-5bvn krb5kdc[11865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991982, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:46:22.127223 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:22.115280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:37479 (local address 127.1.155.190:45907)
0620 21:46:22.115427 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:22.115432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:22.116280 (+ 848us) server_negotiation.cc:407] Connection header received
0620 21:46:22.117552 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:22.117555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:22.117602 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:22.117678 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:22.118858 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.119419 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.121023 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.121251 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.121363 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:22.123808 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:22.123811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:22.123814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:46:22.123845 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:22.125199 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:22.125703 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:22.125708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:22.125710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:22.125797 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:22.126173 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:22.126178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:22.126180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:22.126356 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:22.126461 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:22.126941 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 21:46:22.127061 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:46:22.128120 12371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:22.115692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:45907 (local address 127.1.155.131:37479)
0620 21:46:22.116121 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:22.116161 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:22.117356 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:22.117905 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:22.117912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:22.118263 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:46:22.118727 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:22.118741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.119599 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.119606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:22.120901 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:22.120911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.121397 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.121401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:22.121584 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:22.122215 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:46:22.122232 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:46:22.123636 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:46:22.125321 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:46:22.125327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:46:22.125342 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:46:22.125596 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:46:22.125927 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:46:22.125931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:46:22.125933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:46:22.126073 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:46:22.126458 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:46:22.126466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:46:22.126710 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 21:46:22.126902 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 21:46:22.127232 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
I20260620 21:46:22.129269 12369 heartbeater.cc:347] Connected to a master server at 127.1.155.190:45907
I20260620 21:46:22.129482 12369 heartbeater.cc:464] Registering TS with master...
I20260620 21:46:22.129985 12369 heartbeater.cc:511] Master 127.1.155.190:45907 requested a full tablet report, sending...
I20260620 21:46:22.131059 11910 ts_manager.cc:194] Registered new tserver with Master: 9b151a4dbd464f9aa0f933126f0bd095 (127.1.155.131:34473)
I20260620 21:46:22.131644 11910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:37479
I20260620 21:46:22.133777 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:46:22.145922 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:22.135861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.1.155.190:45907)
0620 21:46:22.136050 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:22.136056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:22.136245 (+ 189us) server_negotiation.cc:407] Connection header received
0620 21:46:22.136332 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:22.136336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:22.136399 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:22.136506 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:46:22.137239 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.137768 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.139287 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.139506 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.139596 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:22.141558 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:46:22.141564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:46:22.141567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:46:22.141597 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:46:22.143609 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:22.144278 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:22.144283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:22.144285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:22.144384 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:46:22.144721 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:46:22.144726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:46:22.144727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:46:22.145108 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:46:22.145255 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:22.145505 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 21:46:22.145696 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 21:46:22.149580 11910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:46:22.152572 11910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:46:22.167847 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:22.162069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.1.155.131:34473)
0620 21:46:22.163261 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:22.163272 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:22.163294 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:46:22.163361 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:22.163365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:22.163521 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:22.163657 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:46:22.163900 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.164672 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.166258 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.167088 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.167219 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:22.167373 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:22.167607 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 21:46:22.167708 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1092,"thread_start_us":124,"threads_started":1}
I20260620 21:46:22.167840 12380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:22.161743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:34473 (local address 127.0.0.1:54724)
0620 21:46:22.162568 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:22.162621 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:22.162747 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:22.163652 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:22.163657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:22.163677 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:46:22.163762 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:22.163768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.164810 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.164817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:22.166120 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:22.166130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.167223 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.167230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:22.167338 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:22.167374 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 21:46:22.167482 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 21:46:22.167683 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":69,"thread_start_us":278,"threads_started":1}
I20260620 21:46:22.168129 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:22.161674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:35277 (local address 127.0.0.1:56082)
0620 21:46:22.162696 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:22.162710 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:22.162804 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:22.163399 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:22.163404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:22.163426 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:46:22.163577 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:22.163584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.165117 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.165121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:22.166483 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:22.166493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.167506 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.167510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:22.167597 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:22.167647 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 21:46:22.167716 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 21:46:22.167787 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":49,"thread_start_us":96,"threads_started":1}
I20260620 21:46:22.168422 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:22.161798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.1.155.130:35277)
0620 21:46:22.162502 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:22.162516 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:22.162904 (+ 388us) server_negotiation.cc:407] Connection header received
0620 21:46:22.163005 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:22.163011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:22.163238 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:22.163400 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:46:22.163740 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.164963 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.166618 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.167400 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.167495 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:22.167618 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:22.167743 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 21:46:22.167832 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":156,"threads_started":1}
I20260620 21:46:22.168490 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:22.162257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:35047 (local address 127.0.0.1:56698)
0620 21:46:22.162973 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:22.162985 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:22.163062 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:22.163736 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:22.163739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:22.163756 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:46:22.163890 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:22.163896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.165229 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.165232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:22.166469 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:22.166477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.167480 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.167484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:22.167587 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:22.168221 (+ 634us) client_negotiation.cc:769] Sending connection context
0620 21:46:22.168303 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 21:46:22.168370 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":700,"thread_start_us":92,"threads_started":1}
I20260620 21:46:22.168880 12385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:22.162658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.1.155.129:35047)
0620 21:46:22.163028 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:22.163043 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:22.163065 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:46:22.163212 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:22.163218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:22.163415 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:22.163559 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:46:22.164021 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.165097 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.166586 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.167136 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.167279 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:22.167416 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:22.168622 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0620 21:46:22.168732 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":91,"threads_started":1}
I20260620 21:46:22.171348 12303 tablet_service.cc:1467] Processing CreateTablet for tablet c057b956bf6c458594d46965e55df070 (DEFAULT_TABLE table=test-table [id=747bf137e51d4f7cbf8802eb8df57ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:46:22.171352 12169 tablet_service.cc:1467] Processing CreateTablet for tablet c057b956bf6c458594d46965e55df070 (DEFAULT_TABLE table=test-table [id=747bf137e51d4f7cbf8802eb8df57ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:46:22.172294 12169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c057b956bf6c458594d46965e55df070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:46:22.172344 12303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c057b956bf6c458594d46965e55df070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:46:22.174572 12035 tablet_service.cc:1467] Processing CreateTablet for tablet c057b956bf6c458594d46965e55df070 (DEFAULT_TABLE table=test-table [id=747bf137e51d4f7cbf8802eb8df57ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:46:22.175606 12035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c057b956bf6c458594d46965e55df070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:46:22.178727 12386 tablet_bootstrap.cc:492] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095: Bootstrap starting.
I20260620 21:46:22.180094 12387 tablet_bootstrap.cc:492] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d: Bootstrap starting.
I20260620 21:46:22.180855 12386 tablet_bootstrap.cc:654] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095: Neither blocks nor log segments found. Creating new log.
I20260620 21:46:22.182031 12386 log.cc:824] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095: Log is configured to *not* fsync() on all Append() calls
I20260620 21:46:22.182600 12388 tablet_bootstrap.cc:492] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395: Bootstrap starting.
I20260620 21:46:22.182670 12387 tablet_bootstrap.cc:654] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d: Neither blocks nor log segments found. Creating new log.
I20260620 21:46:22.183804 12387 log.cc:824] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:46:22.184170 12386 tablet_bootstrap.cc:492] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095: No bootstrap required, opened a new log
I20260620 21:46:22.184379 12386 ts_tablet_manager.cc:1397] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 21:46:22.184806 12388 tablet_bootstrap.cc:654] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395: Neither blocks nor log segments found. Creating new log.
I20260620 21:46:22.185582 12387 tablet_bootstrap.cc:492] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d: No bootstrap required, opened a new log
I20260620 21:46:22.185786 12387 ts_tablet_manager.cc:1397] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:46:22.186095 12388 log.cc:824] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395: Log is configured to *not* fsync() on all Append() calls
I20260620 21:46:22.188403 12388 tablet_bootstrap.cc:492] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395: No bootstrap required, opened a new log
I20260620 21:46:22.188596 12388 ts_tablet_manager.cc:1397] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:46:22.188560 12386 raft_consensus.cc:348] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a3e1a635304d2aa0a8432f7050e10d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35277 } } peers { permanent_uuid: "9b151a4dbd464f9aa0f933126f0bd095" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34473 } } peers { permanent_uuid: "da48b3be20954949b190bedfee030395" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 35047 } }
I20260620 21:46:22.188846 12386 raft_consensus.cc:374] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:46:22.188913 12386 raft_consensus.cc:729] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b151a4dbd464f9aa0f933126f0bd095, State: Initialized, Role: FOLLOWER
I20260620 21:46:22.189172 12387 raft_consensus.cc:348] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a3e1a635304d2aa0a8432f7050e10d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35277 } } peers { permanent_uuid: "9b151a4dbd464f9aa0f933126f0bd095" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34473 } } peers { permanent_uuid: "da48b3be20954949b190bedfee030395" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 35047 } }
I20260620 21:46:22.189399 12387 raft_consensus.cc:374] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:46:22.189452 12387 raft_consensus.cc:729] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97a3e1a635304d2aa0a8432f7050e10d, State: Initialized, Role: FOLLOWER
I20260620 21:46:22.189393 12386 consensus_queue.cc:260] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a3e1a635304d2aa0a8432f7050e10d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35277 } } peers { permanent_uuid: "9b151a4dbd464f9aa0f933126f0bd095" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34473 } } peers { permanent_uuid: "da48b3be20954949b190bedfee030395" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 35047 } }
I20260620 21:46:22.189888 12387 consensus_queue.cc:260] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a3e1a635304d2aa0a8432f7050e10d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35277 } } peers { permanent_uuid: "9b151a4dbd464f9aa0f933126f0bd095" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34473 } } peers { permanent_uuid: "da48b3be20954949b190bedfee030395" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 35047 } }
I20260620 21:46:22.190773 12235 heartbeater.cc:503] Master 127.1.155.190:45907 was elected leader, sending a full tablet report...
I20260620 21:46:22.190788 12386 ts_tablet_manager.cc:1428] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260620 21:46:22.190824 12387 ts_tablet_manager.cc:1428] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:46:22.191105 12369 heartbeater.cc:503] Master 127.1.155.190:45907 was elected leader, sending a full tablet report...
I20260620 21:46:22.193974 12388 raft_consensus.cc:348] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a3e1a635304d2aa0a8432f7050e10d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35277 } } peers { permanent_uuid: "9b151a4dbd464f9aa0f933126f0bd095" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34473 } } peers { permanent_uuid: "da48b3be20954949b190bedfee030395" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 35047 } }
I20260620 21:46:22.194416 12388 raft_consensus.cc:374] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:46:22.194518 12388 raft_consensus.cc:729] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da48b3be20954949b190bedfee030395, State: Initialized, Role: FOLLOWER
I20260620 21:46:22.194983 12388 consensus_queue.cc:260] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a3e1a635304d2aa0a8432f7050e10d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35277 } } peers { permanent_uuid: "9b151a4dbd464f9aa0f933126f0bd095" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34473 } } peers { permanent_uuid: "da48b3be20954949b190bedfee030395" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 35047 } }
I20260620 21:46:22.195767 12101 heartbeater.cc:503] Master 127.1.155.190:45907 was elected leader, sending a full tablet report...
I20260620 21:46:22.196111 12388 ts_tablet_manager.cc:1428] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 21:46:22.327603 12394 raft_consensus.cc:482] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:46:22.327873 12394 raft_consensus.cc:504] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a3e1a635304d2aa0a8432f7050e10d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35277 } } peers { permanent_uuid: "9b151a4dbd464f9aa0f933126f0bd095" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34473 } } peers { permanent_uuid: "da48b3be20954949b190bedfee030395" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 35047 } }
I20260620 21:46:22.329032 12394 leader_election.cc:288] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97a3e1a635304d2aa0a8432f7050e10d (127.1.155.130:35277), 9b151a4dbd464f9aa0f933126f0bd095 (127.1.155.131:34473)
I20260620 21:46:22.334622 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:22.329469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:33107 (local address 127.1.155.131:34473)
0620 21:46:22.329611 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:22.329614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:22.329859 (+ 245us) server_negotiation.cc:407] Connection header received
0620 21:46:22.329971 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:22.329974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:22.330025 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:22.330112 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:46:22.330539 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.331324 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.333275 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.333931 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.334061 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:22.334112 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:22.334416 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 21:46:22.334488 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:46:22.334642 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:22.329364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:36123 (local address 127.1.155.130:35277)
0620 21:46:22.329511 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:22.329518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:22.329807 (+ 289us) server_negotiation.cc:407] Connection header received
0620 21:46:22.329971 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:22.329974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:22.330028 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:22.330114 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:46:22.330539 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.331320 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.333275 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.333930 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.334062 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:22.334112 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:22.334416 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 21:46:22.334487 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 21:46:22.334748 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:22.329242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:35277 (local address 127.1.155.129:36123)
0620 21:46:22.329697 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:22.329712 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:22.329858 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:22.330143 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:22.330146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:22.330184 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:46:22.330414 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:22.330421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.331447 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.331451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:22.333116 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:22.333128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.334085 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.334090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:22.334244 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:22.334300 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 21:46:22.334385 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 21:46:22.334490 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":294,"spinlock_wait_cycles":6400,"thread_start_us":142,"threads_started":1}
I20260620 21:46:22.334750 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:22.329353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:34473 (local address 127.1.155.129:33107)
0620 21:46:22.329761 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:22.329774 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:46:22.329871 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:46:22.330143 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:46:22.330146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:46:22.330211 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:46:22.330414 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:22.330421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.331447 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.331451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:22.333123 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:46:22.333129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.334085 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.334090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:46:22.334258 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:22.334313 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 21:46:22.334386 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 21:46:22.334490 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":324,"spinlock_wait_cycles":28160,"thread_start_us":92,"threads_started":1}
I20260620 21:46:22.335419 12189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c057b956bf6c458594d46965e55df070" candidate_uuid: "da48b3be20954949b190bedfee030395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97a3e1a635304d2aa0a8432f7050e10d" is_pre_election: true
I20260620 21:46:22.335414 12323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c057b956bf6c458594d46965e55df070" candidate_uuid: "da48b3be20954949b190bedfee030395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b151a4dbd464f9aa0f933126f0bd095" is_pre_election: true
I20260620 21:46:22.335721 12323 raft_consensus.cc:2454] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da48b3be20954949b190bedfee030395 in term 0.
I20260620 21:46:22.335745 12189 raft_consensus.cc:2454] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da48b3be20954949b190bedfee030395 in term 0.
I20260620 21:46:22.336212 11990 leader_election.cc:302] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97a3e1a635304d2aa0a8432f7050e10d, da48b3be20954949b190bedfee030395; no voters:
I20260620 21:46:22.336493 12394 raft_consensus.cc:2781] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:46:22.336575 12394 raft_consensus.cc:482] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:46:22.336617 12394 raft_consensus.cc:3037] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:46:22.337769 12394 raft_consensus.cc:504] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a3e1a635304d2aa0a8432f7050e10d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35277 } } peers { permanent_uuid: "9b151a4dbd464f9aa0f933126f0bd095" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34473 } } peers { permanent_uuid: "da48b3be20954949b190bedfee030395" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 35047 } }
I20260620 21:46:22.338290 12394 leader_election.cc:288] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [CANDIDATE]: Term 1 election: Requested vote from peers 97a3e1a635304d2aa0a8432f7050e10d (127.1.155.130:35277), 9b151a4dbd464f9aa0f933126f0bd095 (127.1.155.131:34473)
I20260620 21:46:22.338651 12323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c057b956bf6c458594d46965e55df070" candidate_uuid: "da48b3be20954949b190bedfee030395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b151a4dbd464f9aa0f933126f0bd095"
I20260620 21:46:22.338649 12189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c057b956bf6c458594d46965e55df070" candidate_uuid: "da48b3be20954949b190bedfee030395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97a3e1a635304d2aa0a8432f7050e10d"
I20260620 21:46:22.338794 12323 raft_consensus.cc:3037] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:46:22.338807 12189 raft_consensus.cc:3037] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:46:22.339810 12189 raft_consensus.cc:2454] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da48b3be20954949b190bedfee030395 in term 1.
I20260620 21:46:22.339990 12323 raft_consensus.cc:2454] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da48b3be20954949b190bedfee030395 in term 1.
I20260620 21:46:22.340111 11990 leader_election.cc:302] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97a3e1a635304d2aa0a8432f7050e10d, da48b3be20954949b190bedfee030395; no voters:
I20260620 21:46:22.340313 12394 raft_consensus.cc:2781] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:46:22.340543 12394 raft_consensus.cc:686] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [term 1 LEADER]: Becoming Leader. State: Replica: da48b3be20954949b190bedfee030395, State: Running, Role: LEADER
I20260620 21:46:22.340924 12394 consensus_queue.cc:237] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a3e1a635304d2aa0a8432f7050e10d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35277 } } peers { permanent_uuid: "9b151a4dbd464f9aa0f933126f0bd095" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34473 } } peers { permanent_uuid: "da48b3be20954949b190bedfee030395" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 35047 } }
I20260620 21:46:22.344148 11909 catalog_manager.cc:5495] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 reported cstate change: term changed from 0 to 1, leader changed from <none> to da48b3be20954949b190bedfee030395 (127.1.155.129). New cstate: current_term: 1 leader_uuid: "da48b3be20954949b190bedfee030395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a3e1a635304d2aa0a8432f7050e10d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b151a4dbd464f9aa0f933126f0bd095" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da48b3be20954949b190bedfee030395" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 35047 } health_report { overall_health: HEALTHY } } }
I20260620 21:46:22.364382 12385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:22.360965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.1.155.129:35047)
0620 21:46:22.361075 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 21:46:22.361079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:46:22.361091 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:46:22.361189 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:46:22.361192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:46:22.361242 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:46:22.361329 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:46:22.361535 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.362119 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.362952 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:46:22.363183 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:46:22.363261 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:46:22.363585 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:46:22.363981 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:46:22.364129 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 21:46:22.364184 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 21:46:22.364232 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 21:46:22.364423 12370 tablet.cc:2369] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:46:22.370081 12323 raft_consensus.cc:1261] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095 [term 1 FOLLOWER]: Refusing update from remote peer da48b3be20954949b190bedfee030395: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:46:22.370241 12189 raft_consensus.cc:1261] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d [term 1 FOLLOWER]: Refusing update from remote peer da48b3be20954949b190bedfee030395: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:46:22.370792 12394 consensus_queue.cc:1035] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b151a4dbd464f9aa0f933126f0bd095" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:46:22.370980 12397 consensus_queue.cc:1035] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a3e1a635304d2aa0a8432f7050e10d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 35277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:46:22.379019 12400 mvcc.cc:205] Tried to move back new op lower bound from 7299039159780532224 to 7299039159671099392. Current Snapshot: MvccSnapshot[applied={T|T < 7299039159780532224}]
I20260620 21:46:22.394119 11910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:46:22.394304 11910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:46:22.396663 11910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5f0b5319e5d4e11ac60e782ee58ce31: Sending DeleteTablet for 3 replicas of tablet c057b956bf6c458594d46965e55df070
I20260620 21:46:22.397568 12303 tablet_service.cc:1514] Processing DeleteTablet for tablet c057b956bf6c458594d46965e55df070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:46:22 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:54724
I20260620 21:46:22.397568 12169 tablet_service.cc:1514] Processing DeleteTablet for tablet c057b956bf6c458594d46965e55df070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:46:22 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:56082
I20260620 21:46:22.397702 12035 tablet_service.cc:1514] Processing DeleteTablet for tablet c057b956bf6c458594d46965e55df070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:46:22 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:56698
I20260620 21:46:22.398244 12411 tablet_replica.cc:330] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395: stopping tablet replica
I20260620 21:46:22.398269 12410 tablet_replica.cc:330] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d: stopping tablet replica
I20260620 21:46:22.398556 12411 raft_consensus.cc:2229] T c057b956bf6c458594d46965e55df070 P da48b3be20954949b190bedfee030395 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:46:22.398592 12410 raft_consensus.cc:2229] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:46:22.398860 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 11972
I20260620 21:46:22.400434 12410 raft_consensus.cc:2258] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:46:22.400780 12409 tablet_replica.cc:330] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095: stopping tablet replica
I20260620 21:46:22.401062 12409 raft_consensus.cc:2229] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:46:22.401345 12409 raft_consensus.cc:2258] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:46:22.401971 12410 ts_tablet_manager.cc:1905] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:46:22.402683 12409 ts_tablet_manager.cc:1905] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:46:22.404158 12410 ts_tablet_manager.cc:1918] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:46:22.404322 12410 log.cc:1196] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/c057b956bf6c458594d46965e55df070
I20260620 21:46:22.404640 12410 ts_tablet_manager.cc:1939] T c057b956bf6c458594d46965e55df070 P 97a3e1a635304d2aa0a8432f7050e10d: Deleting consensus metadata
I20260620 21:46:22.405076 12409 ts_tablet_manager.cc:1918] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:46:22.405167 12409 log.cc:1196] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/c057b956bf6c458594d46965e55df070
I20260620 21:46:22.405483 12409 ts_tablet_manager.cc:1939] T c057b956bf6c458594d46965e55df070 P 9b151a4dbd464f9aa0f933126f0bd095: Deleting consensus metadata
I20260620 21:46:22.405969 11896 catalog_manager.cc:4841] TS 97a3e1a635304d2aa0a8432f7050e10d (127.1.155.130:35277): tablet c057b956bf6c458594d46965e55df070 (table test-table [id=747bf137e51d4f7cbf8802eb8df57ba7]) successfully deleted
W20260620 21:46:22.406092 11895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.155.129:35047 (error 108)
I20260620 21:46:22.406641 11894 catalog_manager.cc:4841] TS 9b151a4dbd464f9aa0f933126f0bd095 (127.1.155.131:34473): tablet c057b956bf6c458594d46965e55df070 (table test-table [id=747bf137e51d4f7cbf8802eb8df57ba7]) successfully deleted
I20260620 21:46:22.406893 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 12106
I20260620 21:46:22.407119 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:46:22.406812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:35047 (local address 127.0.0.1:56722)
0620 21:46:22.406933 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 21:46:22.407026 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 21:46:22.407981 11895 catalog_manager.cc:4584] TS da48b3be20954949b190bedfee030395 (127.1.155.129:35047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c057b956bf6c458594d46965e55df070: Network error: Client connection negotiation failed: client connection to 127.1.155.129:35047: connect: Connection refused (error 111)
I20260620 21:46:22.412585 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 12240
I20260620 21:46:22.418193 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 11878
2026-06-20T21:46:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2420 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:46:22 dist-test-slave-5bvn krb5kdc[12415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:46:22 dist-test-slave-5bvn krb5kdc[12415](info): set up 2 sockets
Jun 20 21:46:22 dist-test-slave-5bvn krb5kdc[12415](info): commencing operation
krb5kdc: starting...
W20260620 21:46:23.459043 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:46:23 dist-test-slave-5bvn krb5kdc[12415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781991983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:46:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:46:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.155.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.155.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/rangeradmin_127.1.155.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/rangeradmin_127.1.155.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.155.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.155.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/rangerlookup_127.1.155.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/rangerlookup_127.1.155.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.155.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.155.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/HTTP_127.1.155.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/HTTP_127.1.155.148@KRBTEST.COM.keytab.
I20260620 21:46:23.628479 1646 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskeoPTss/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/postgres -l logfile start
2026-06-20 21:46:26.171 UTC [12445] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 21:46:26.171 UTC [12445] LOG: listening on IPv4 address "127.1.155.148", port 48473
2026-06-20 21:46:26.175 UTC [12445] LOG: listening on Unix socket "/tmp/.s.PGSQL.48473"
2026-06-20 21:46:26.197 UTC [12448] LOG: database system was shut down at 2026-06-20 21:46:24 UTC
2026-06-20 21:46:26.202 UTC [12445] LOG: database system is ready to accept connections
I20260620 21:46:27.150745 1646 mini_postgres.cc:96] Postgres bound to 48473
2026-06-20 21:46:27.157 UTC [12456] FATAL: database "slave" does not exist
127.1.155.148:48473 - accepting connections
I20260620 21:46:27.157975 1646 mini_ranger.cc:161] Starting Ranger...
I20260620 21:46:27.173166 1646 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 21:46:27.667627 1646 mini_ranger.cc:87] Created ranger Postgres database
I20260620 21:46:27.667740 1646 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-admin
2026-06-20 21:46:27,956 [I] DB FLAVOR :POSTGRES
2026-06-20 21:46:27,956 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 21:46:27,956 [I] Checking connection..
2026-06-20 21:46:27,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 21:46:28,226 [I] Checking connection passed.
2026-06-20 21:46:28,226 [I] --------- Verifying version history table ---------
2026-06-20 21:46:28,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:46:28,474 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 21:46:28,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:46:28,705 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 21:46:28,705 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 21:46:28,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 21:46:28.960 UTC [12520] WARNING: there is no transaction in progress
2026-06-20 21:46:28,966 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 21:46:28,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:46:29,223 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 21:46:29,223 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 21:46:29,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 21:46:29,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 21:46:29,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5bvn.c.gcp-upstream.internal','N') ;"
2026-06-20 21:46:29,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 21:46:30,193 [I] Table x_portal_user does not exist in database ranger
2026-06-20 21:46:30,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 21:46:30,430 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 21:46:30,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 21:46:30,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 21:46:30,914 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 21:46:30,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 21:46:31.910 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:31.928 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:31.946 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:31.964 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:31.983 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:32.002 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:32.064 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:32.072 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:32.092 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:32.104 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:32.117 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:32.132 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:32.145 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:32.152 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:32.160 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:32.167 UTC [12628] WARNING: there is no transaction in progress
2026-06-20 21:46:32,829 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 21:46:32,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 21:46:33,075 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 21:46:33,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 21:46:33,314 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 21:46:33,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 21:46:33,552 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 21:46:33,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 21:46:33,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 21:46:34,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5bvn.c.gcp-upstream.internal';"
2026-06-20 21:46:34,253 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 21:46:34,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 21:46:34,485 [I] DB_PATCHES have already been applied
I20260620 21:46:34.491093 1646 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-admin:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskeoPTss/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 21:46:34.491184 1646 mini_ranger.cc:193] Using host: 127.1.155.148
I20260620 21:46:34.494624 1646 mini_ranger.cc:227] Ranger admin URL: http://127.1.155.148:51109
Jun 20, 2026 9:46:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:46:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:46:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 9:46:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 9:46:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 9:46:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 9:46:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 21:46:35.613365 1646 mini_ranger.cc:160] Time spent starting Ranger: real 8.455s user 0.005s sys 0.005s
Jun 20, 2026 9:46:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 9:46:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 9:46:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 9:46:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 9:46:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 9:46:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 9:46:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 9:46:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 9:46:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 9:46:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 9:46:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 9:46:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 9:46:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 9:46:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 9:46:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 9:46:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 9:46:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 9:46:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 21:46:55.942824 1646 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.155.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.155.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/rangerkms_127.1.155.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/rangerkms_127.1.155.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.155.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.155.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.155.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/HTTP_127.1.155.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.155.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/HTTP_127.1.155.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 21:46:56 dist-test-slave-5bvn krb5kdc[12415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992016, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 21:46:56.034904 1646 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 21:46:56.045089 1646 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 21:46:56.946404 1646 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 21:46:56.946501 1646 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-kms
I20260620 21:46:56.946604 1646 mini_ranger_kms.cc:227] Using postgres at 127.1.155.148:48473
2026-06-20 21:46:57,142 [I] DB FLAVOR :POSTGRES
2026-06-20 21:46:57,142 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 21:46:57,142 [I] Checking connection
2026-06-20 21:46:57,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 21:46:57,392 [I] connection success
2026-06-20 21:46:57,392 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 21:46:57,393 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 21:46:57,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 21:46:57,624 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 21:46:57,625 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 21:46:57,625 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 21:46:57,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:48473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 21:46:57,895 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 21:46:58.330118 1646 mini_ranger_kms.cc:310] Created kms service
I20260620 21:46:58.507241 1646 mini_ranger_kms.cc:326] Created kudu user
I20260620 21:46:58.553300 1646 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 21:46:58 dist-test-slave-5bvn krb5kdc[12415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992016, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:46:58.945084 1646 mini_ranger_kms.cc:384] Added ranger policy
I20260620 21:46:58.945242 1646 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-kms:/tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 21:46:58.945282 1646 mini_ranger_kms.cc:240] Using host: 127.1.155.148
I20260620 21:46:58.948745 1646 mini_ranger_kms.cc:276] Ranger KMS PID: 12808
I20260620 21:46:58.948884 1646 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.155.148:50475
Jun 20, 2026 9:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 9:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 9:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 9:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 21:47:01.209237 1646 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.174s user 0.008s sys 0.173s
I20260620 21:47:01.209381 1646 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 21:47:01.209515 1646 mini_ranger_kms.cc:401] 127.1.155.148:50475/kms/v1/keys
Jun 20 21:47:01 dist-test-slave-5bvn krb5kdc[12415](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992021, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.155.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 21:47:01 dist-test-slave-5bvn krb5kdc[12415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992021, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.155.148@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
Jun 20 21:47:02 dist-test-slave-5bvn krb5kdc[12415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992021, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.155.148@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
Jun 20 21:47:04 dist-test-slave-5bvn krb5kdc[12415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:47:04.368432 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:41861
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:41861
--ranger_config_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.155.148:50475/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:47:04.478974 12859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:47:04.479302 12859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:47:04.479391 12859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:47:04.483506 12859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:47:04.483596 12859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:47:04.483636 12859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:47:04.483660 12859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:47:04.483685 12859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:47:04.489164 12859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.155.148:50475/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:41861
--ranger_config_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:41861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:47:04.490332 12859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:47:04.491358 12859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:47:04.500643 12864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:04.500713 12865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:04.500733 12867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:04.501056 12859 server_base.cc:1034] running on GCE node
I20260620 21:47:04.501610 12859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:47:04.502945 12859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:47:04.504173 12859 hybrid_clock.cc:648] HybridClock initialized: now 1781992024504146 us; error 46 us; skew 500 ppm
Jun 20 21:47:04 dist-test-slave-5bvn krb5kdc[12415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:04.507218 12859 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:47:04.508219 12859 webserver.cc:460] Webserver started at http://127.1.155.190:44513/ using document root <none> and password file <none>
I20260620 21:47:04.508775 12859 fs_manager.cc:359] Metadata directory not provided
I20260620 21:47:04.508849 12859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:47:04.509087 12859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:47:04 dist-test-slave-5bvn krb5kdc[12415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:04.563396 12859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "4fd7467e014d40af9a5de8ec306294f3"
format_stamp: "Formatted at 2026-06-20 21:47:04 on dist-test-slave-5bvn"
server_key: "63160d6e306eda2ae9fa9a97d13c8ee0"
server_key_iv: "ab936932eef193110c522a049fe820eb"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:04.564078 12859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "4fd7467e014d40af9a5de8ec306294f3"
format_stamp: "Formatted at 2026-06-20 21:47:04 on dist-test-slave-5bvn"
server_key: "63160d6e306eda2ae9fa9a97d13c8ee0"
server_key_iv: "ab936932eef193110c522a049fe820eb"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:04.567981 12859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:47:04.581589 12874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:04.583026 12859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:47:04.583174 12859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "4fd7467e014d40af9a5de8ec306294f3"
format_stamp: "Formatted at 2026-06-20 21:47:04 on dist-test-slave-5bvn"
server_key: "63160d6e306eda2ae9fa9a97d13c8ee0"
server_key_iv: "ab936932eef193110c522a049fe820eb"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:04.583318 12859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:47:04.600319 12859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:47:04.603374 12859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:47:04.603590 12859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:47:04.612669 12859 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:41861
I20260620 21:47:04.612717 12926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:41861 every 8 connection(s)
I20260620 21:47:04.613783 12859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:47:04.616081 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 12859
I20260620 21:47:04.616235 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:47:04.616492 1646 external_mini_cluster.cc:1428] Setting key 493c27441a44f000c3d0b0bdfb16a4ca
I20260620 21:47:04.617604 12927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 21:47:04 dist-test-slave-5bvn krb5kdc[12415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:47:04.624984 12927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3: Bootstrap starting.
I20260620 21:47:04.627750 12927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3: Neither blocks nor log segments found. Creating new log.
I20260620 21:47:04.629057 12927 log.cc:824] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3: Log is configured to *not* fsync() on all Append() calls
I20260620 21:47:04.630237 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:04.617916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54502 (local address 127.1.155.190:41861)
0620 21:47:04.618321 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:04.618344 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:04.618378 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:47:04.619075 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:04.619091 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:04.619403 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:04.619742 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:04.620511 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:04.621493 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:04.622417 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:04.622880 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:04.623047 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:04.625539 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:04.625544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:04.625557 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:04.625583 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:04.627221 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:04.627677 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:04.627684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:04.627695 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:04.627794 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:04.628111 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:04.628113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:04.628114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:04.628482 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:04.628687 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:04.628959 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 21:47:04.629266 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":137,"threads_started":1}
I20260620 21:47:04.631470 12927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3: No bootstrap required, opened a new log
I20260620 21:47:04.634876 12927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd7467e014d40af9a5de8ec306294f3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 41861 } }
I20260620 21:47:04.635087 12927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:47:04.635152 12927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd7467e014d40af9a5de8ec306294f3, State: Initialized, Role: FOLLOWER
I20260620 21:47:04.635569 12927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd7467e014d40af9a5de8ec306294f3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 41861 } }
I20260620 21:47:04.635684 12927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:47:04.635775 12927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:47:04.635869 12927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:47:04.636778 12927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd7467e014d40af9a5de8ec306294f3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 41861 } }
I20260620 21:47:04.637115 12927 leader_election.cc:302] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fd7467e014d40af9a5de8ec306294f3; no voters:
I20260620 21:47:04.637463 12927 leader_election.cc:288] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:47:04.637552 12932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:47:04.637737 12932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [term 1 LEADER]: Becoming Leader. State: Replica: 4fd7467e014d40af9a5de8ec306294f3, State: Running, Role: LEADER
I20260620 21:47:04.638049 12932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd7467e014d40af9a5de8ec306294f3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 41861 } }
I20260620 21:47:04.638739 12927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:47:04.639691 12933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fd7467e014d40af9a5de8ec306294f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd7467e014d40af9a5de8ec306294f3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 41861 } } }
I20260620 21:47:04.639816 12933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [sys.catalog]: This master's current role is: LEADER
I20260620 21:47:04.640177 12934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fd7467e014d40af9a5de8ec306294f3. Latest consensus state: current_term: 1 leader_uuid: "4fd7467e014d40af9a5de8ec306294f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd7467e014d40af9a5de8ec306294f3" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 41861 } } }
I20260620 21:47:04.640225 12941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:47:04.640323 12934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3 [sys.catalog]: This master's current role is: LEADER
I20260620 21:47:04.643472 12941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:47:04.643910 12927 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 21:47:04.655344 12941 catalog_manager.cc:1347] Generated new cluster ID: a3a9306a6f3747c4898993df36205141
I20260620 21:47:04.655519 12941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:47:04.670830 12941 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:47:04.671900 12941 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:47:04.682396 12941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3: Generated new TSK 0
I20260620 21:47:04.683249 12941 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:47:05.381366 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.155.148:50475/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:41861
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:47:05.541549 12972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:47:05.541958 12972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:47:05.542025 12972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:47:05.547188 12972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:47:05.547266 12972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:47:05.547328 12972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:47:05.554267 12972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.155.148:50475/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:41861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:47:05.555655 12972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:47:05.556929 12972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:47:05.569182 12980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:05.569211 12979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:05.569484 12972 server_base.cc:1034] running on GCE node
W20260620 21:47:05.569684 12982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:05.570230 12972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:47:05.571079 12972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:47:05.572293 12972 hybrid_clock.cc:648] HybridClock initialized: now 1781992025572256 us; error 46 us; skew 500 ppm
Jun 20 21:47:05 dist-test-slave-5bvn krb5kdc[12415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:05.575672 12972 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:47:05.576679 12972 webserver.cc:460] Webserver started at http://127.1.155.129:41737/ using document root <none> and password file <none>
I20260620 21:47:05.577404 12972 fs_manager.cc:359] Metadata directory not provided
I20260620 21:47:05.577512 12972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:47:05.577749 12972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:47:05 dist-test-slave-5bvn krb5kdc[12415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:05.595642 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "037157bf654e41709a1fbba6297ee440"
format_stamp: "Formatted at 2026-06-20 21:47:05 on dist-test-slave-5bvn"
server_key: "1325221e85d7780f0f50bf6ad63c6a00"
server_key_iv: "bcb4ebd66631d4a46b2ca4001113cd7a"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:05.596513 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "037157bf654e41709a1fbba6297ee440"
format_stamp: "Formatted at 2026-06-20 21:47:05 on dist-test-slave-5bvn"
server_key: "1325221e85d7780f0f50bf6ad63c6a00"
server_key_iv: "bcb4ebd66631d4a46b2ca4001113cd7a"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:05.601640 12972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 21:47:05.623225 12989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:05.624890 12972 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 21:47:05.625022 12972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "037157bf654e41709a1fbba6297ee440"
format_stamp: "Formatted at 2026-06-20 21:47:05 on dist-test-slave-5bvn"
server_key: "1325221e85d7780f0f50bf6ad63c6a00"
server_key_iv: "bcb4ebd66631d4a46b2ca4001113cd7a"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:05.625149 12972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:47:05.647610 12972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:47:05.651250 12972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:47:05.651511 12972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:47:05.652205 12972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:47:05.653242 12972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:47:05.653340 12972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:05.653470 12972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:47:05.653548 12972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:05.666195 12972 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:33123
I20260620 21:47:05.666260 13102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:33123 every 8 connection(s)
I20260620 21:47:05.667457 12972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:47:05.670666 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 12972
I20260620 21:47:05.670807 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:47:05.671146 1646 external_mini_cluster.cc:1428] Setting key 390f0834affd5225257a9540fc16402a
Jun 20 21:47:05 dist-test-slave-5bvn krb5kdc[12415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:47:05.688571 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:05.670443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:56211 (local address 127.1.155.190:41861)
0620 21:47:05.671766 (+ 1323us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:05.671775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:05.675008 (+ 3233us) server_negotiation.cc:407] Connection header received
0620 21:47:05.676262 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:05.676265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:05.676320 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:05.676379 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:05.678107 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:05.678968 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:05.680655 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:05.681031 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:05.681106 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:05.684190 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:05.684194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:05.684197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:05.684223 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:05.686054 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:05.686705 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:05.686708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:05.686710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:05.686801 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:05.687201 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:05.687204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:05.687206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:05.687438 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:05.687517 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:05.688258 (+ 741us) server_negotiation.cc:299] Negotiation successful
0620 21:47:05.688387 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1238,"thread_start_us":94,"threads_started":1}
I20260620 21:47:05.690418 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:05.670884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:41861 (local address 127.1.155.129:56211)
0620 21:47:05.674816 (+ 3932us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:05.674892 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:05.676003 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:05.676817 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:05.676826 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:05.677198 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:05.677927 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:05.677947 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:05.679125 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:05.679129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:05.680519 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:05.680529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:05.681317 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:05.681321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:05.681525 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:05.682261 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:05.682281 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:05.684000 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:05.686189 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:05.686195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:05.686214 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:05.686546 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:05.686907 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:05.686910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:05.686912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:05.687092 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:05.687581 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:05.687587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:05.687982 (+ 395us) client_negotiation.cc:769] Sending connection context
0620 21:47:05.689029 (+ 1047us) client_negotiation.cc:240] Negotiation successful
0620 21:47:05.689369 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3673,"thread_start_us":146,"threads_started":1}
I20260620 21:47:05.692056 13104 heartbeater.cc:347] Connected to a master server at 127.1.155.190:41861
I20260620 21:47:05.692298 13104 heartbeater.cc:464] Registering TS with master...
I20260620 21:47:05.692867 13104 heartbeater.cc:511] Master 127.1.155.190:41861 requested a full tablet report, sending...
I20260620 21:47:05.694765 12891 ts_manager.cc:194] Registered new tserver with Master: 037157bf654e41709a1fbba6297ee440 (127.1.155.129:33123)
I20260620 21:47:05.696264 12891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:56211
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:47:05.724448 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.155.148:50475/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:41861
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:47:05 dist-test-slave-5bvn krb5kdc[12415](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 21:47:05.882719 13111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:47:05.883168 13111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:47:05.883276 13111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:47:05.889356 13111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:47:05.889518 13111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:47:05.889657 13111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:47:05.897718 13111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.155.148:50475/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:41861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:47:05.899176 13111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:47:05.900492 13111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:47:05.913201 13118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:05.913424 13117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:05.913713 13120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:05.914130 13111 server_base.cc:1034] running on GCE node
I20260620 21:47:05.914573 13111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:47:05.915369 13111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:47:05.916559 13111 hybrid_clock.cc:648] HybridClock initialized: now 1781992025916539 us; error 37 us; skew 500 ppm
Jun 20 21:47:05 dist-test-slave-5bvn krb5kdc[12415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:05.920244 13111 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:47:05.921425 13111 webserver.cc:460] Webserver started at http://127.1.155.130:33571/ using document root <none> and password file <none>
I20260620 21:47:05.922124 13111 fs_manager.cc:359] Metadata directory not provided
I20260620 21:47:05.922195 13111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:47:05.922443 13111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:47:05 dist-test-slave-5bvn krb5kdc[12415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:05.937918 13111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "e158e14eb0fe40b38656c763c1fa51b3"
format_stamp: "Formatted at 2026-06-20 21:47:05 on dist-test-slave-5bvn"
server_key: "0ef8e21ff887aa188e3d86b93db488d3"
server_key_iv: "93faad33f219bf9c777adf2abb26bb44"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:05.938884 13111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "e158e14eb0fe40b38656c763c1fa51b3"
format_stamp: "Formatted at 2026-06-20 21:47:05 on dist-test-slave-5bvn"
server_key: "0ef8e21ff887aa188e3d86b93db488d3"
server_key_iv: "93faad33f219bf9c777adf2abb26bb44"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:05.946025 13111 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260620 21:47:05.961473 13127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:05.963351 13111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:47:05.963583 13111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "e158e14eb0fe40b38656c763c1fa51b3"
format_stamp: "Formatted at 2026-06-20 21:47:05 on dist-test-slave-5bvn"
server_key: "0ef8e21ff887aa188e3d86b93db488d3"
server_key_iv: "93faad33f219bf9c777adf2abb26bb44"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:05.963804 13111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:47:05.980906 13111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:47:05.984798 13111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:47:05.985018 13111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:47:05.985733 13111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:47:05.986852 13111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:47:05.986922 13111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:05.986984 13111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:47:05.987022 13111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:05.998775 13111 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:38345
I20260620 21:47:05.998873 13240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:38345 every 8 connection(s)
I20260620 21:47:05.999748 13111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:47:06.003208 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 13111
I20260620 21:47:06.003381 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:47:06.003720 1646 external_mini_cluster.cc:1428] Setting key 24d2c835d2ad8032a417ac93179ea2f9
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
Jun 20 21:47:06 dist-test-slave-5bvn krb5kdc[12415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:47:06.035713 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.002040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:48965 (local address 127.1.155.190:41861)
0620 21:47:06.002206 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:06.002211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:06.003125 (+ 914us) server_negotiation.cc:407] Connection header received
0620 21:47:06.004512 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:06.004516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:06.004574 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:06.004654 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:06.005977 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.007215 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.020111 (+ 12896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.020543 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.020646 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.030844 (+ 10198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:06.030849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:06.030853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:06.030886 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:06.032901 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:06.033699 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:06.033706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:06.033708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:06.033798 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:06.034339 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:06.034342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:06.034344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:06.034601 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:06.034709 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:06.035391 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 21:47:06.035539 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 21:47:06.036785 13243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.002497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:41861 (local address 127.1.155.130:48965)
0620 21:47:06.002973 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:06.003027 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:06.004273 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:06.004850 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:06.004860 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:06.005163 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:06.005730 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.005747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.007419 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.007424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.019860 (+ 12436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.019883 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.022103 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.022109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.022366 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.028686 (+ 6320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:06.028715 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:06.030517 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:06.033103 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:06.033111 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:06.033130 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:06.033508 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:06.033969 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:06.033973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:06.033975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:06.034207 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:06.034715 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:06.034722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:06.035100 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 21:47:06.035322 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 21:47:06.035687 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:47:06.038174 13241 heartbeater.cc:347] Connected to a master server at 127.1.155.190:41861
I20260620 21:47:06.038424 13241 heartbeater.cc:464] Registering TS with master...
I20260620 21:47:06.039057 13241 heartbeater.cc:511] Master 127.1.155.190:41861 requested a full tablet report, sending...
I20260620 21:47:06.039202 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.155.148:50475/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:41861
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:47:06.040156 12891 ts_manager.cc:194] Registered new tserver with Master: e158e14eb0fe40b38656c763c1fa51b3 (127.1.155.130:38345)
I20260620 21:47:06.040999 12891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:48965
Jun 20 21:47:06 dist-test-slave-5bvn krb5kdc[12415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
W20260620 21:47:06.192258 13246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:47:06.192664 13246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:47:06.192734 13246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:47:06.199875 13246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:47:06.199962 13246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:47:06.200069 13246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:47:06.208763 13246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.155.148:50475/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:41861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:47:06.210320 13246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:47:06.211753 13246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:47:06.231050 13252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:06.231233 13246 server_base.cc:1034] running on GCE node
W20260620 21:47:06.231916 13251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:06.232199 13254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:06.232726 13246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:47:06.233584 13246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:47:06.234763 13246 hybrid_clock.cc:648] HybridClock initialized: now 1781992026234743 us; error 70 us; skew 500 ppm
Jun 20 21:47:06 dist-test-slave-5bvn krb5kdc[12415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992026, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:06.238490 13246 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:47:06.239856 13246 webserver.cc:460] Webserver started at http://127.1.155.131:32925/ using document root <none> and password file <none>
I20260620 21:47:06.240746 13246 fs_manager.cc:359] Metadata directory not provided
I20260620 21:47:06.240844 13246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:47:06.241110 13246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:47:06 dist-test-slave-5bvn krb5kdc[12415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992026, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:06.261791 13246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "d3de67d7ddae4d009d3ea5ecba4396e5"
format_stamp: "Formatted at 2026-06-20 21:47:06 on dist-test-slave-5bvn"
server_key: "087df6588e3efe04bf4b4c3ff3032014"
server_key_iv: "28fc5fdd1551ec8fe237b4878f724009"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:06.275605 13246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "d3de67d7ddae4d009d3ea5ecba4396e5"
format_stamp: "Formatted at 2026-06-20 21:47:06 on dist-test-slave-5bvn"
server_key: "087df6588e3efe04bf4b4c3ff3032014"
server_key_iv: "28fc5fdd1551ec8fe237b4878f724009"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:06.281370 13246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 21:47:06.299163 13261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:06.300885 13246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:47:06.301051 13246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "d3de67d7ddae4d009d3ea5ecba4396e5"
format_stamp: "Formatted at 2026-06-20 21:47:06 on dist-test-slave-5bvn"
server_key: "087df6588e3efe04bf4b4c3ff3032014"
server_key_iv: "28fc5fdd1551ec8fe237b4878f724009"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:06.301203 13246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:47:06.316668 13246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:47:06.320818 13246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:47:06.321036 13246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:47:06.321744 13246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:47:06.322935 13246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:47:06.323001 13246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:06.323051 13246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:47:06.323071 13246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:06.333494 13246 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:44215
I20260620 21:47:06.334625 13246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:47:06.341007 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 13246
I20260620 21:47:06.341130 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:47:06.341403 1646 external_mini_cluster.cc:1428] Setting key 2257dc72a414d42e95616615d9290a3e
I20260620 21:47:06.337553 13374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:44215 every 8 connection(s)
Jun 20 21:47:06 dist-test-slave-5bvn krb5kdc[12415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992026, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:47:06.358343 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.341766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:56929 (local address 127.1.155.190:41861)
0620 21:47:06.342105 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:06.342109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:06.344567 (+ 2458us) server_negotiation.cc:407] Connection header received
0620 21:47:06.345784 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:06.345787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:06.345873 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:06.345926 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:06.347686 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.348547 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.350429 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.350786 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.350906 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.353941 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:06.353945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:06.353948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:06.353976 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:06.355792 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:06.356462 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:06.356466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:06.356468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:06.356562 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:06.357012 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:06.357016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:06.357017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:06.357257 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:06.357367 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:06.358060 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 21:47:06.358188 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260620 21:47:06.359952 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.339799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:41861 (local address 127.1.155.131:56929)
0620 21:47:06.344408 (+ 4609us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:06.344474 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:06.345537 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:06.346355 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:06.346364 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:06.346751 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:06.347415 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.347434 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.348722 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.348726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.350300 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.350312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.350974 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.350978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.351186 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.351948 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:06.351970 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:06.353705 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:06.355935 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:06.355941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:06.355960 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:06.356314 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:06.356678 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:06.356681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:06.356683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:06.356873 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:06.357380 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:06.357388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:06.357725 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 21:47:06.358563 (+ 838us) client_negotiation.cc:240] Negotiation successful
0620 21:47:06.358923 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4369,"thread_start_us":160,"threads_started":1}
I20260620 21:47:06.361153 13375 heartbeater.cc:347] Connected to a master server at 127.1.155.190:41861
I20260620 21:47:06.361399 13375 heartbeater.cc:464] Registering TS with master...
I20260620 21:47:06.361981 13375 heartbeater.cc:511] Master 127.1.155.190:41861 requested a full tablet report, sending...
I20260620 21:47:06.363234 12891 ts_manager.cc:194] Registered new tserver with Master: d3de67d7ddae4d009d3ea5ecba4396e5 (127.1.155.131:44215)
I20260620 21:47:06.363986 12891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:56929
I20260620 21:47:06.367871 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:47:06.379688 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.370387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.1.155.190:41861)
0620 21:47:06.370501 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:06.370506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:06.370774 (+ 268us) server_negotiation.cc:407] Connection header received
0620 21:47:06.370896 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:06.370899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:06.370947 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:06.371034 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:06.371896 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.372714 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.373972 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.374310 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.374416 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.376111 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:06.376115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:06.376118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:06.376142 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:06.377924 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:06.378411 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:06.378415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:06.378417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:06.378504 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:06.378812 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:06.378815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:06.378817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:06.379025 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:06.379148 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:06.379417 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 21:47:06.379532 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:47:06.383394 12891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:47:06.387709 12891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:47:06.407038 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.397372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:38345 (local address 127.0.0.1:58572)
0620 21:47:06.399861 (+ 2489us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:06.399931 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:06.400063 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:06.400769 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:06.400776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:06.400798 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:47:06.400935 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.400943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.403195 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.403199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.404411 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.404421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.406454 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.406458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.406587 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.406630 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 21:47:06.406745 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 21:47:06.406864 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2326,"mutex_wait_us":5,"thread_start_us":92,"threads_started":1}
I20260620 21:47:06.407660 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.398061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58572 (local address 127.1.155.130:38345)
0620 21:47:06.399630 (+ 1569us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:06.399640 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:06.400044 (+ 404us) server_negotiation.cc:407] Connection header received
0620 21:47:06.400222 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:06.400229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:06.400442 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:06.400594 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:47:06.401080 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.403057 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.404573 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.406122 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.406232 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.406386 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:06.407411 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0620 21:47:06.407511 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1479,"thread_start_us":106,"threads_started":1}
I20260620 21:47:06.407806 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.398504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.1.155.131:44215)
0620 21:47:06.399446 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:06.399459 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:06.400443 (+ 984us) server_negotiation.cc:407] Connection header received
0620 21:47:06.400518 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:06.400523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:06.400695 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:06.400821 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:47:06.401434 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.402613 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.406130 (+ 3517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.406858 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.406968 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.407090 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:06.407592 (+ 502us) server_negotiation.cc:299] Negotiation successful
0620 21:47:06.407683 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":80,"threads_started":1}
I20260620 21:47:06.407502 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.397457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:44215 (local address 127.0.0.1:46158)
0620 21:47:06.400278 (+ 2821us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:06.400290 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:06.400357 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:06.401212 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:06.401215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:06.401225 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:47:06.401320 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.401325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.402738 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.402742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.405984 (+ 3242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.405995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.407147 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.407150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.407245 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.407264 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 21:47:06.407327 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 21:47:06.407386 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2765,"mutex_wait_us":1863,"thread_start_us":92,"threads_started":1}
I20260620 21:47:06.411767 13175 tablet_service.cc:1467] Processing CreateTablet for tablet 737967acb5654de883174a960d547a8f (DEFAULT_TABLE table=test-table [id=5c92e6d9a01c45a8b6df176060eede43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:47:06.412931 13175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 737967acb5654de883174a960d547a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:47:06.417768 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 737967acb5654de883174a960d547a8f (DEFAULT_TABLE table=test-table [id=5c92e6d9a01c45a8b6df176060eede43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:47:06.418727 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.398425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:33123 (local address 127.0.0.1:42240)
0620 21:47:06.401043 (+ 2618us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:06.401055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:06.401130 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:06.404823 (+ 3693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:06.404826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:06.404842 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:47:06.404944 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.404949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.415592 (+ 10643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.415596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.416852 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.416864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.418228 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.418232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.418384 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.418412 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:47:06.418492 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 21:47:06.418574 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2560,"thread_start_us":80,"threads_started":1}
I20260620 21:47:06.418905 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 737967acb5654de883174a960d547a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:47:06.419370 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.398801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.1.155.129:33123)
0620 21:47:06.404241 (+ 5440us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:06.404248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:06.404271 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:47:06.404340 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:06.404345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:06.404525 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:06.404656 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:47:06.405066 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.415434 (+ 10368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.417022 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.417880 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.417998 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.418157 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:06.419121 (+ 964us) server_negotiation.cc:299] Negotiation successful
0620 21:47:06.419225 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5331,"thread_start_us":83,"threads_started":1}
I20260620 21:47:06.422315 13037 tablet_service.cc:1467] Processing CreateTablet for tablet 737967acb5654de883174a960d547a8f (DEFAULT_TABLE table=test-table [id=5c92e6d9a01c45a8b6df176060eede43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:47:06.423384 13037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 737967acb5654de883174a960d547a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:47:06.432073 13393 tablet_bootstrap.cc:492] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440: Bootstrap starting.
I20260620 21:47:06.433349 13395 tablet_bootstrap.cc:492] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5: Bootstrap starting.
I20260620 21:47:06.434773 13393 tablet_bootstrap.cc:654] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440: Neither blocks nor log segments found. Creating new log.
I20260620 21:47:06.435710 13393 log.cc:824] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440: Log is configured to *not* fsync() on all Append() calls
I20260620 21:47:06.436044 13395 tablet_bootstrap.cc:654] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5: Neither blocks nor log segments found. Creating new log.
I20260620 21:47:06.436833 13395 log.cc:824] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5: Log is configured to *not* fsync() on all Append() calls
I20260620 21:47:06.437486 13394 tablet_bootstrap.cc:492] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3: Bootstrap starting.
I20260620 21:47:06.437968 13393 tablet_bootstrap.cc:492] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440: No bootstrap required, opened a new log
I20260620 21:47:06.438145 13393 ts_tablet_manager.cc:1397] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:47:06.440217 13394 tablet_bootstrap.cc:654] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3: Neither blocks nor log segments found. Creating new log.
I20260620 21:47:06.441033 13394 log.cc:824] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3: Log is configured to *not* fsync() on all Append() calls
I20260620 21:47:06.442750 13393 raft_consensus.cc:348] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e158e14eb0fe40b38656c763c1fa51b3" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 38345 } } peers { permanent_uuid: "d3de67d7ddae4d009d3ea5ecba4396e5" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44215 } } peers { permanent_uuid: "037157bf654e41709a1fbba6297ee440" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 33123 } }
I20260620 21:47:06.443014 13393 raft_consensus.cc:374] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:47:06.443063 13393 raft_consensus.cc:729] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037157bf654e41709a1fbba6297ee440, State: Initialized, Role: FOLLOWER
I20260620 21:47:06.443181 13395 tablet_bootstrap.cc:492] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5: No bootstrap required, opened a new log
I20260620 21:47:06.443364 13395 ts_tablet_manager.cc:1397] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260620 21:47:06.443840 13393 consensus_queue.cc:260] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e158e14eb0fe40b38656c763c1fa51b3" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 38345 } } peers { permanent_uuid: "d3de67d7ddae4d009d3ea5ecba4396e5" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44215 } } peers { permanent_uuid: "037157bf654e41709a1fbba6297ee440" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 33123 } }
I20260620 21:47:06.444980 13393 ts_tablet_manager.cc:1428] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 21:47:06.445387 13104 heartbeater.cc:503] Master 127.1.155.190:41861 was elected leader, sending a full tablet report...
I20260620 21:47:06.447208 13395 raft_consensus.cc:348] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e158e14eb0fe40b38656c763c1fa51b3" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 38345 } } peers { permanent_uuid: "d3de67d7ddae4d009d3ea5ecba4396e5" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44215 } } peers { permanent_uuid: "037157bf654e41709a1fbba6297ee440" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 33123 } }
I20260620 21:47:06.447453 13395 raft_consensus.cc:374] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:47:06.447521 13395 raft_consensus.cc:729] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3de67d7ddae4d009d3ea5ecba4396e5, State: Initialized, Role: FOLLOWER
I20260620 21:47:06.448026 13395 consensus_queue.cc:260] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e158e14eb0fe40b38656c763c1fa51b3" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 38345 } } peers { permanent_uuid: "d3de67d7ddae4d009d3ea5ecba4396e5" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44215 } } peers { permanent_uuid: "037157bf654e41709a1fbba6297ee440" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 33123 } }
I20260620 21:47:06.448972 13395 ts_tablet_manager.cc:1428] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:47:06.449272 13375 heartbeater.cc:503] Master 127.1.155.190:41861 was elected leader, sending a full tablet report...
I20260620 21:47:06.452720 13394 tablet_bootstrap.cc:492] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3: No bootstrap required, opened a new log
I20260620 21:47:06.452955 13394 ts_tablet_manager.cc:1397] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260620 21:47:06.456998 13394 raft_consensus.cc:348] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e158e14eb0fe40b38656c763c1fa51b3" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 38345 } } peers { permanent_uuid: "d3de67d7ddae4d009d3ea5ecba4396e5" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44215 } } peers { permanent_uuid: "037157bf654e41709a1fbba6297ee440" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 33123 } }
I20260620 21:47:06.457307 13394 raft_consensus.cc:374] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:47:06.457365 13394 raft_consensus.cc:729] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e158e14eb0fe40b38656c763c1fa51b3, State: Initialized, Role: FOLLOWER
I20260620 21:47:06.457908 13394 consensus_queue.cc:260] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e158e14eb0fe40b38656c763c1fa51b3" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 38345 } } peers { permanent_uuid: "d3de67d7ddae4d009d3ea5ecba4396e5" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44215 } } peers { permanent_uuid: "037157bf654e41709a1fbba6297ee440" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 33123 } }
I20260620 21:47:06.459239 13394 ts_tablet_manager.cc:1428] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:47:06.459630 13241 heartbeater.cc:503] Master 127.1.155.190:41861 was elected leader, sending a full tablet report...
Jun 20 21:47:06 dist-test-slave-5bvn krb5kdc[12415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
W20260620 21:47:06.500946 13242 tablet.cc:2369] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:47:06.594390 13376 tablet.cc:2369] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:47:06.660776 13401 raft_consensus.cc:482] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:47:06.661051 13401 raft_consensus.cc:504] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e158e14eb0fe40b38656c763c1fa51b3" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 38345 } } peers { permanent_uuid: "d3de67d7ddae4d009d3ea5ecba4396e5" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44215 } } peers { permanent_uuid: "037157bf654e41709a1fbba6297ee440" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 33123 } }
I20260620 21:47:06.662617 13401 leader_election.cc:288] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3de67d7ddae4d009d3ea5ecba4396e5 (127.1.155.131:44215), 037157bf654e41709a1fbba6297ee440 (127.1.155.129:33123)
W20260620 21:47:06.669394 13105 tablet.cc:2369] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:47:06.675161 13399 raft_consensus.cc:482] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:47:06.675335 13399 raft_consensus.cc:504] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e158e14eb0fe40b38656c763c1fa51b3" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 38345 } } peers { permanent_uuid: "d3de67d7ddae4d009d3ea5ecba4396e5" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44215 } } peers { permanent_uuid: "037157bf654e41709a1fbba6297ee440" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 33123 } }
I20260620 21:47:06.676564 13399 leader_election.cc:288] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e158e14eb0fe40b38656c763c1fa51b3 (127.1.155.130:38345), d3de67d7ddae4d009d3ea5ecba4396e5 (127.1.155.131:44215)
I20260620 21:47:06.679464 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.663220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:36953 (local address 127.1.155.129:33123)
0620 21:47:06.663384 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:06.663389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:06.670140 (+ 6751us) server_negotiation.cc:407] Connection header received
0620 21:47:06.670367 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:06.670371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:06.670433 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:06.670483 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:47:06.670890 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.671718 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.677992 (+ 6274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.678698 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.678800 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.678853 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:06.679237 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 21:47:06.679320 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:47:06.680091 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.663107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:33123 (local address 127.1.155.130:36953)
0620 21:47:06.670011 (+ 6904us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:06.670033 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:06.670236 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:06.670629 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:06.670632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:06.670650 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:47:06.670764 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.670770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.671873 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.671877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.673297 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.673308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.678965 (+ 5657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.678969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.679086 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.679111 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:47:06.679842 (+ 731us) client_negotiation.cc:240] Negotiation successful
0620 21:47:06.679932 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6797,"thread_start_us":111,"threads_started":1}
I20260620 21:47:06.680876 13046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737967acb5654de883174a960d547a8f" candidate_uuid: "e158e14eb0fe40b38656c763c1fa51b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037157bf654e41709a1fbba6297ee440" is_pre_election: true
I20260620 21:47:06.681231 13046 raft_consensus.cc:2454] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e158e14eb0fe40b38656c763c1fa51b3 in term 0.
I20260620 21:47:06.682015 13128 leader_election.cc:302] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037157bf654e41709a1fbba6297ee440, e158e14eb0fe40b38656c763c1fa51b3; no voters:
I20260620 21:47:06.682340 13401 raft_consensus.cc:2781] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:47:06.682412 13401 raft_consensus.cc:482] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:47:06.682448 13401 raft_consensus.cc:3037] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:47:06.683794 13401 raft_consensus.cc:504] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e158e14eb0fe40b38656c763c1fa51b3" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 38345 } } peers { permanent_uuid: "d3de67d7ddae4d009d3ea5ecba4396e5" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44215 } } peers { permanent_uuid: "037157bf654e41709a1fbba6297ee440" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 33123 } }
I20260620 21:47:06.684180 13401 leader_election.cc:288] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [CANDIDATE]: Term 1 election: Requested vote from peers d3de67d7ddae4d009d3ea5ecba4396e5 (127.1.155.131:44215), 037157bf654e41709a1fbba6297ee440 (127.1.155.129:33123)
I20260620 21:47:06.684789 13046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737967acb5654de883174a960d547a8f" candidate_uuid: "e158e14eb0fe40b38656c763c1fa51b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037157bf654e41709a1fbba6297ee440"
I20260620 21:47:06.684916 13046 raft_consensus.cc:3037] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:47:06.685814 13046 raft_consensus.cc:2454] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e158e14eb0fe40b38656c763c1fa51b3 in term 1.
I20260620 21:47:06.686250 13128 leader_election.cc:302] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037157bf654e41709a1fbba6297ee440, e158e14eb0fe40b38656c763c1fa51b3; no voters:
I20260620 21:47:06.686491 13401 raft_consensus.cc:2781] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:47:06.686699 13401 raft_consensus.cc:686] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [term 1 LEADER]: Becoming Leader. State: Replica: e158e14eb0fe40b38656c763c1fa51b3, State: Running, Role: LEADER
I20260620 21:47:06.687067 13401 consensus_queue.cc:237] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e158e14eb0fe40b38656c763c1fa51b3" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 38345 } } peers { permanent_uuid: "d3de67d7ddae4d009d3ea5ecba4396e5" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44215 } } peers { permanent_uuid: "037157bf654e41709a1fbba6297ee440" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 33123 } }
I20260620 21:47:06.690474 12891 catalog_manager.cc:5495] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e158e14eb0fe40b38656c763c1fa51b3 (127.1.155.130). New cstate: current_term: 1 leader_uuid: "e158e14eb0fe40b38656c763c1fa51b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e158e14eb0fe40b38656c763c1fa51b3" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 38345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3de67d7ddae4d009d3ea5ecba4396e5" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "037157bf654e41709a1fbba6297ee440" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 33123 } health_report { overall_health: UNKNOWN } } }
I20260620 21:47:06.695935 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.663691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:49867 (local address 127.1.155.131:44215)
0620 21:47:06.663855 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:06.663859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:06.682133 (+ 18274us) server_negotiation.cc:407] Connection header received
0620 21:47:06.682260 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:06.682263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:06.682334 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:06.682426 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:47:06.690267 (+ 7841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.691044 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.694602 (+ 3558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.695195 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.695325 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.695379 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:06.695732 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 21:47:06.695810 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:47:06.696311 13404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.663556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:44215 (local address 127.1.155.130:49867)
0620 21:47:06.682037 (+ 18481us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:06.682055 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:06.682163 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:06.689914 (+ 7751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:06.689919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:06.689949 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:47:06.690098 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.690106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.691228 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.691231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.694453 (+ 3222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.694463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.695495 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.695498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.695595 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.695616 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:47:06.696079 (+ 463us) client_negotiation.cc:240] Negotiation successful
0620 21:47:06.696159 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18395,"thread_start_us":89,"threads_started":1}
I20260620 21:47:06.697156 13329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737967acb5654de883174a960d547a8f" candidate_uuid: "e158e14eb0fe40b38656c763c1fa51b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3de67d7ddae4d009d3ea5ecba4396e5" is_pre_election: true
I20260620 21:47:06.697479 13329 raft_consensus.cc:2454] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e158e14eb0fe40b38656c763c1fa51b3 in term 0.
I20260620 21:47:06.697867 13328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737967acb5654de883174a960d547a8f" candidate_uuid: "e158e14eb0fe40b38656c763c1fa51b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3de67d7ddae4d009d3ea5ecba4396e5"
I20260620 21:47:06.697978 13328 raft_consensus.cc:3037] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:47:06.699329 13328 raft_consensus.cc:2454] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e158e14eb0fe40b38656c763c1fa51b3 in term 1.
I20260620 21:47:06.705250 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.677216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:39875 (local address 127.1.155.130:38345)
0620 21:47:06.677353 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:06.677359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:06.692485 (+ 15126us) server_negotiation.cc:407] Connection header received
0620 21:47:06.692923 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:06.692926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:06.692974 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:06.693082 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:47:06.694692 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.695397 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.703757 (+ 8360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.704403 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.704504 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.704571 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:06.705036 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 21:47:06.705121 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:47:06.707451 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.676889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:38345 (local address 127.1.155.129:39875)
0620 21:47:06.692351 (+ 15462us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:06.692367 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:06.692478 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:06.694398 (+ 1920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:06.694401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:06.694422 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:47:06.694554 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.694560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.695544 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.695547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.703597 (+ 8050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.703609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.704700 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.704704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.704879 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.704906 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:47:06.707192 (+ 2286us) client_negotiation.cc:240] Negotiation successful
0620 21:47:06.707286 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15382,"thread_start_us":96,"threads_started":1}
I20260620 21:47:06.709316 13179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737967acb5654de883174a960d547a8f" candidate_uuid: "037157bf654e41709a1fbba6297ee440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e158e14eb0fe40b38656c763c1fa51b3" is_pre_election: true
I20260620 21:47:06.710619 13408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.677988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:41291 (local address 127.1.155.131:44215)
0620 21:47:06.684389 (+ 6401us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:06.684398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:06.695468 (+ 11070us) server_negotiation.cc:407] Connection header received
0620 21:47:06.695524 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:06.695528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:06.695578 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:06.695658 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:47:06.700411 (+ 4753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.701184 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.707722 (+ 6538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.708357 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.708459 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.708508 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:06.710377 (+ 1869us) server_negotiation.cc:299] Negotiation successful
0620 21:47:06.710463 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6320,"thread_start_us":139,"threads_started":1}
I20260620 21:47:06.711180 13407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.677591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:44215 (local address 127.1.155.129:41291)
0620 21:47:06.692674 (+ 15083us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:06.692687 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:06.692778 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:06.696275 (+ 3497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:06.696278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:06.696299 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:47:06.700264 (+ 3965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.700273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.701366 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.701369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.702671 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:06.702683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.708607 (+ 5924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.708610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:06.708712 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.708734 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:47:06.710956 (+ 2222us) client_negotiation.cc:240] Negotiation successful
0620 21:47:06.711042 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15021,"thread_start_us":117,"threads_started":1}
I20260620 21:47:06.711745 13328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737967acb5654de883174a960d547a8f" candidate_uuid: "037157bf654e41709a1fbba6297ee440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3de67d7ddae4d009d3ea5ecba4396e5" is_pre_election: true
I20260620 21:47:06.711966 13328 raft_consensus.cc:2379] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 037157bf654e41709a1fbba6297ee440 in current term 1: Already voted for candidate e158e14eb0fe40b38656c763c1fa51b3 in this term.
I20260620 21:47:06.712534 12993 leader_election.cc:302] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 037157bf654e41709a1fbba6297ee440; no voters: d3de67d7ddae4d009d3ea5ecba4396e5, e158e14eb0fe40b38656c763c1fa51b3
I20260620 21:47:06.713047 13399 raft_consensus.cc:2726] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 21:47:06.781734 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.776467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.1.155.130:38345)
0620 21:47:06.776617 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:06.776622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:06.776640 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:47:06.776743 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:06.776747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:06.776820 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:06.776971 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:47:06.777371 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.778176 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.779435 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:06.779770 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:06.779866 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:06.780310 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:47:06.780946 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:47:06.781079 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:06.781537 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 21:47:06.781600 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 21:47:06.790238 13046 raft_consensus.cc:1261] T 737967acb5654de883174a960d547a8f P 037157bf654e41709a1fbba6297ee440 [term 1 FOLLOWER]: Refusing update from remote peer e158e14eb0fe40b38656c763c1fa51b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:47:06.791085 13409 consensus_queue.cc:1035] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "037157bf654e41709a1fbba6297ee440" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:47:06.799031 13328 raft_consensus.cc:1261] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5 [term 1 FOLLOWER]: Refusing update from remote peer e158e14eb0fe40b38656c763c1fa51b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:47:06.799782 13409 consensus_queue.cc:1035] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3de67d7ddae4d009d3ea5ecba4396e5" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 44215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:47:06.813473 13418 mvcc.cc:205] Tried to move back new op lower bound from 7299039341722341376 to 7299039341314043904. Current Snapshot: MvccSnapshot[applied={T|T < 7299039341722341376}]
I20260620 21:47:06.814477 13414 mvcc.cc:205] Tried to move back new op lower bound from 7299039341722341376 to 7299039341314043904. Current Snapshot: MvccSnapshot[applied={T|T < 7299039341722341376}]
I20260620 21:47:06.824836 12890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:47:06.825028 12890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:47:06.827392 12890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fd7467e014d40af9a5de8ec306294f3: Sending DeleteTablet for 3 replicas of tablet 737967acb5654de883174a960d547a8f
I20260620 21:47:06.828828 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 12972
I20260620 21:47:06.828888 13175 tablet_service.cc:1514] Processing DeleteTablet for tablet 737967acb5654de883174a960d547a8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:47:06 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:58572
I20260620 21:47:06.829375 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 737967acb5654de883174a960d547a8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:47:06 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:46158
I20260620 21:47:06.830197 13429 tablet_replica.cc:330] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5: stopping tablet replica
I20260620 21:47:06.830547 13429 raft_consensus.cc:2229] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:47:06.830893 13429 raft_consensus.cc:2258] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:47:06.831276 13428 tablet_replica.cc:330] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3: stopping tablet replica
I20260620 21:47:06.831585 13428 raft_consensus.cc:2229] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:47:06.831947 13428 raft_consensus.cc:2258] T 737967acb5654de883174a960d547a8f P e158e14eb0fe40b38656c763c1fa51b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:47:06.833310 13429 ts_tablet_manager.cc:1905] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:47:06.836652 13429 ts_tablet_manager.cc:1918] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:47:06.836755 13429 log.cc:1196] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/737967acb5654de883174a960d547a8f
I20260620 21:47:06.837054 13429 ts_tablet_manager.cc:1939] T 737967acb5654de883174a960d547a8f P d3de67d7ddae4d009d3ea5ecba4396e5: Deleting consensus metadata
W20260620 21:47:06.837567 12875 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.155.129:33123 (error 108)
I20260620 21:47:06.838379 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 13111
I20260620 21:47:06.838341 12878 catalog_manager.cc:4841] TS d3de67d7ddae4d009d3ea5ecba4396e5 (127.1.155.131:44215): tablet 737967acb5654de883174a960d547a8f (table test-table [id=5c92e6d9a01c45a8b6df176060eede43]) successfully deleted
I20260620 21:47:06.839900 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.839579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:33123 (local address 127.0.0.1:42244)
0620 21:47:06.839759 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:06.839820 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:33123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260620 21:47:06.840099 12875 catalog_manager.cc:4584] TS 037157bf654e41709a1fbba6297ee440 (127.1.155.129:33123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 737967acb5654de883174a960d547a8f: Network error: Client connection negotiation failed: client connection to 127.1.155.129:33123: connect: Connection refused (error 111)
I20260620 21:47:06.844702 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 13246
I20260620 21:47:06.844869 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:06.844592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:38345 (local address 127.0.0.1:58580)
0620 21:47:06.844749 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:06.844799 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.130:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 21:47:06.850310 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 12859
I20260620 21:47:06.859165 1646 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 21:47:07.205411 1646 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 21:47:07.205569 1646 mini_ranger.cc:66] Stopping Ranger...
I20260620 21:47:07.551115 1646 mini_ranger.cc:68] Stopped Ranger
2026-06-20 21:47:07.551 UTC [12445] LOG: received smart shutdown request
2026-06-20 21:47:07.554 UTC [12445] LOG: background worker "logical replication launcher" (PID 12454) exited with exit code 1
2026-06-20 21:47:07.555 UTC [12449] LOG: shutting down
2026-06-20 21:47:07.725 UTC [12445] LOG: database system is shut down
2026-06-20T21:47:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45430 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:47:07 dist-test-slave-5bvn krb5kdc[13439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:47:07 dist-test-slave-5bvn krb5kdc[13439](info): set up 2 sockets
Jun 20 21:47:07 dist-test-slave-5bvn krb5kdc[13439](info): commencing operation
krb5kdc: starting...
W20260620 21:47:08.922284 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:47:09 dist-test-slave-5bvn krb5kdc[13439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:47:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.155.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.155.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/rangeradmin_127.1.155.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/rangeradmin_127.1.155.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.155.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.155.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/rangerlookup_127.1.155.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/rangerlookup_127.1.155.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.155.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.155.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/HTTP_127.1.155.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/HTTP_127.1.155.148@KRBTEST.COM.keytab.
I20260620 21:47:09.113147 1646 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskeoPTss/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/postgres -l logfile start
2026-06-20 21:47:11.764 UTC [13469] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 21:47:11.764 UTC [13469] LOG: listening on IPv4 address "127.1.155.148", port 33437
2026-06-20 21:47:11.767 UTC [13469] LOG: listening on Unix socket "/tmp/.s.PGSQL.33437"
2026-06-20 21:47:11.802 UTC [13472] LOG: database system was shut down at 2026-06-20 21:47:10 UTC
2026-06-20 21:47:11.806 UTC [13469] LOG: database system is ready to accept connections
I20260620 21:47:12.785962 1646 mini_postgres.cc:96] Postgres bound to 33437
2026-06-20 21:47:12.792 UTC [13480] FATAL: database "slave" does not exist
127.1.155.148:33437 - accepting connections
I20260620 21:47:12.792960 1646 mini_ranger.cc:161] Starting Ranger...
I20260620 21:47:12.807659 1646 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 21:47:13.300978 1646 mini_ranger.cc:87] Created ranger Postgres database
I20260620 21:47:13.301071 1646 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-admin
2026-06-20 21:47:13,580 [I] DB FLAVOR :POSTGRES
2026-06-20 21:47:13,580 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 21:47:13,580 [I] Checking connection..
2026-06-20 21:47:13,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 21:47:13,820 [I] Checking connection passed.
2026-06-20 21:47:13,821 [I] --------- Verifying version history table ---------
2026-06-20 21:47:13,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:47:14,065 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 21:47:14,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:47:14,319 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 21:47:14,319 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 21:47:14,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 21:47:14.586 UTC [13544] WARNING: there is no transaction in progress
2026-06-20 21:47:14,592 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 21:47:14,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:47:14,859 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 21:47:14,859 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 21:47:14,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 21:47:15,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 21:47:15,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5bvn.c.gcp-upstream.internal','N') ;"
2026-06-20 21:47:15,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 21:47:15,884 [I] Table x_portal_user does not exist in database ranger
2026-06-20 21:47:15,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 21:47:16,141 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 21:47:16,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 21:47:16,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 21:47:16,615 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 21:47:16,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 21:47:17.565 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:17.581 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:17.599 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:17.615 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:17.631 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:17.649 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:17.702 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:17.709 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:17.724 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:17.734 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:17.745 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:17.759 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:17.769 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:17.775 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:17.782 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:17.789 UTC [13652] WARNING: there is no transaction in progress
2026-06-20 21:47:18,423 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 21:47:18,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 21:47:18,659 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 21:47:18,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 21:47:18,923 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 21:47:18,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 21:47:19,165 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 21:47:19,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 21:47:19,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 21:47:19,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5bvn.c.gcp-upstream.internal';"
2026-06-20 21:47:19,883 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 21:47:19,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 21:47:20,125 [I] DB_PATCHES have already been applied
I20260620 21:47:20.131172 1646 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-admin:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskeoPTss/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 21:47:20.131245 1646 mini_ranger.cc:193] Using host: 127.1.155.148
I20260620 21:47:20.134826 1646 mini_ranger.cc:227] Ranger admin URL: http://127.1.155.148:36997
Jun 20, 2026 9:47:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:47:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:47:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 9:47:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 9:47:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 9:47:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 9:47:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 21:47:21.248085 1646 mini_ranger.cc:160] Time spent starting Ranger: real 8.455s user 0.005s sys 0.005s
Jun 20, 2026 9:47:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 9:47:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 9:47:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 9:47:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 9:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 9:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 9:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 9:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 9:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 9:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 9:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 9:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 9:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 9:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 9:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 9:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 9:47:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 9:47:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 21:47:41.778321 1646 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.155.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.155.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/rangerkms_127.1.155.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.155.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/rangerkms_127.1.155.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.155.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.155.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.155.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/HTTP_127.1.155.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.155.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/HTTP_127.1.155.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 21:47:41 dist-test-slave-5bvn krb5kdc[13439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992061, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 21:47:41.868011 1646 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 21:47:41.877097 1646 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 21:47:42.798729 1646 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 21:47:42.798805 1646 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-kms
I20260620 21:47:42.798820 1646 mini_ranger_kms.cc:227] Using postgres at 127.1.155.148:33437
2026-06-20 21:47:42,970 [I] DB FLAVOR :POSTGRES
2026-06-20 21:47:42,971 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 21:47:42,971 [I] Checking connection
2026-06-20 21:47:42,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 21:47:43,204 [I] connection success
2026-06-20 21:47:43,205 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 21:47:43,205 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 21:47:43,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 21:47:43,447 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 21:47:43,448 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 21:47:43,448 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 21:47:43,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.155.148:33437/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 21:47:43,699 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 21:47:44.168784 1646 mini_ranger_kms.cc:310] Created kms service
I20260620 21:47:44.379753 1646 mini_ranger_kms.cc:326] Created kudu user
I20260620 21:47:44.425562 1646 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 21:47:44 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992061, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:44.850152 1646 mini_ranger_kms.cc:384] Added ranger policy
I20260620 21:47:44.850311 1646 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-kms:/tmp/dist-test-taskeoPTss/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskeoPTss/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 21:47:44.850338 1646 mini_ranger_kms.cc:240] Using host: 127.1.155.148
I20260620 21:47:44.854174 1646 mini_ranger_kms.cc:276] Ranger KMS PID: 13832
I20260620 21:47:44.854266 1646 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.155.148:47021
Jun 20, 2026 9:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 9:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 9:47:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 9:47:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 21:47:46 dist-test-slave-5bvn krb5kdc[13439](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992066, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.155.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:47.194298 1646 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.326s user 0.014s sys 0.142s
I20260620 21:47:47.194427 1646 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 21:47:47.194494 1646 mini_ranger_kms.cc:401] 127.1.155.148:47021/kms/v1/keys
Jun 20 21:47:47 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992066, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.155.148@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
Jun 20 21:47:47 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992066, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.155.148@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
Jun 20 21:47:50 dist-test-slave-5bvn krb5kdc[13439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:47:50.158311 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:42873
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:42873
--ranger_config_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.155.148:47021/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:47:50.276104 13883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:47:50.276360 13883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:47:50.276400 13883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:47:50.283901 13883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:47:50.283987 13883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:47:50.284034 13883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:47:50.284102 13883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:47:50.284138 13883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:47:50.290930 13883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.155.148:47021/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:42873
--ranger_config_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:42873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:47:50.292589 13883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:47:50.294137 13883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:47:50.304325 13891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:50.304373 13889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:50.304347 13888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:50.305485 13883 server_base.cc:1034] running on GCE node
I20260620 21:47:50.306250 13883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:47:50.307756 13883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:47:50.308976 13883 hybrid_clock.cc:648] HybridClock initialized: now 1781992070308948 us; error 49 us; skew 500 ppm
Jun 20 21:47:50 dist-test-slave-5bvn krb5kdc[13439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992070, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:50.317942 13883 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:47:50.318989 13883 webserver.cc:460] Webserver started at http://127.1.155.190:44617/ using document root <none> and password file <none>
I20260620 21:47:50.319731 13883 fs_manager.cc:359] Metadata directory not provided
I20260620 21:47:50.319820 13883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:47:50.320106 13883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:47:50 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992070, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:50.366017 13883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "862a830f680d408c8e165cd6bd561322"
format_stamp: "Formatted at 2026-06-20 21:47:50 on dist-test-slave-5bvn"
server_key: "cc7126dd040a562d7565513d48e9f4bc"
server_key_iv: "59fc0f196181dd7ca1e48613c17336aa"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:50.366636 13883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "862a830f680d408c8e165cd6bd561322"
format_stamp: "Formatted at 2026-06-20 21:47:50 on dist-test-slave-5bvn"
server_key: "cc7126dd040a562d7565513d48e9f4bc"
server_key_iv: "59fc0f196181dd7ca1e48613c17336aa"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:50.370481 13883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:47:50.384409 13898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:50.385960 13883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:47:50.386109 13883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "862a830f680d408c8e165cd6bd561322"
format_stamp: "Formatted at 2026-06-20 21:47:50 on dist-test-slave-5bvn"
server_key: "cc7126dd040a562d7565513d48e9f4bc"
server_key_iv: "59fc0f196181dd7ca1e48613c17336aa"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:50.386276 13883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:47:50.408378 13883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:47:50.411510 13883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:47:50.411720 13883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:47:50.420085 13883 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:42873
I20260620 21:47:50.420091 13950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:42873 every 8 connection(s)
I20260620 21:47:50.421078 13883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:47:50.423969 13951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:47:50.426014 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 13883
I20260620 21:47:50.426143 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:47:50.426445 1646 external_mini_cluster.cc:1428] Setting key e65b0cf72e207c075f4f7b1762c3de96
I20260620 21:47:50.431281 13951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322: Bootstrap starting.
I20260620 21:47:50.433688 13951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322: Neither blocks nor log segments found. Creating new log.
I20260620 21:47:50.434619 13951 log.cc:824] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322: Log is configured to *not* fsync() on all Append() calls
I20260620 21:47:50.436582 13951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322: No bootstrap required, opened a new log
Jun 20 21:47:50 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:47:50.441354 13951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a830f680d408c8e165cd6bd561322" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42873 } }
I20260620 21:47:50.441668 13951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:47:50.441731 13951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862a830f680d408c8e165cd6bd561322, State: Initialized, Role: FOLLOWER
I20260620 21:47:50.442291 13951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a830f680d408c8e165cd6bd561322" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42873 } }
I20260620 21:47:50.442456 13951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:47:50.442531 13951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:47:50.442632 13951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:47:50.443822 13951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a830f680d408c8e165cd6bd561322" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42873 } }
I20260620 21:47:50.444188 13951 leader_election.cc:302] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 862a830f680d408c8e165cd6bd561322; no voters:
I20260620 21:47:50.444305 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:50.427960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.1.155.190:42873)
0620 21:47:50.428444 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:50.428463 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:50.428507 (+ 44us) server_negotiation.cc:407] Connection header received
0620 21:47:50.429396 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:50.429419 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:50.429874 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:50.430257 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:50.431144 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:50.432636 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:50.433918 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:50.434623 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:50.434784 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:50.437940 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:50.437947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:50.437964 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:50.437995 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:50.440817 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:50.441359 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:50.441369 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:50.441377 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:50.441514 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:50.441861 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:50.441865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:50.441867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:50.442367 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:50.442583 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:50.443029 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 21:47:50.443307 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":153,"threads_started":1}
I20260620 21:47:50.444525 13951 leader_election.cc:288] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:47:50.444890 13957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:47:50.445129 13957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 1 LEADER]: Becoming Leader. State: Replica: 862a830f680d408c8e165cd6bd561322, State: Running, Role: LEADER
I20260620 21:47:50.445756 13951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:47:50.446100 13957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a830f680d408c8e165cd6bd561322" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42873 } }
I20260620 21:47:50.447455 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 862a830f680d408c8e165cd6bd561322. Latest consensus state: current_term: 1 leader_uuid: "862a830f680d408c8e165cd6bd561322" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a830f680d408c8e165cd6bd561322" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42873 } } }
I20260620 21:47:50.448026 13951 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 21:47:50.448088 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "862a830f680d408c8e165cd6bd561322" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a830f680d408c8e165cd6bd561322" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42873 } } }
I20260620 21:47:50.448320 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [sys.catalog]: This master's current role is: LEADER
I20260620 21:47:50.448822 13970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:47:50.449108 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [sys.catalog]: This master's current role is: LEADER
I20260620 21:47:50.455631 13970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:47:50.463605 13970 catalog_manager.cc:1347] Generated new cluster ID: 3af3066a08ac4584bb92b607be5c5b71
I20260620 21:47:50.463680 13970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:47:50.486931 13970 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:47:50.488219 13970 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:47:50.495410 13970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322: Generated new TSK 0
I20260620 21:47:50.496418 13970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:47:51.222251 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.155.148:47021/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42873
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:47:51.361809 13996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:47:51.362160 13996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:47:51.362233 13996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:47:51.367060 13996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:47:51.367146 13996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:47:51.367260 13996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:47:51.374735 13996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.155.148:47021/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:47:51.376096 13996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:47:51.377379 13996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:47:51.389143 14004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:51.389312 14003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:51.389467 14006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:51.390412 13996 server_base.cc:1034] running on GCE node
I20260620 21:47:51.390925 13996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:47:51.391710 13996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:47:51.392936 13996 hybrid_clock.cc:648] HybridClock initialized: now 1781992071392915 us; error 36 us; skew 500 ppm
Jun 20 21:47:51 dist-test-slave-5bvn krb5kdc[13439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992071, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:51.396406 13996 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:47:51.397317 13996 webserver.cc:460] Webserver started at http://127.1.155.129:38995/ using document root <none> and password file <none>
I20260620 21:47:51.397943 13996 fs_manager.cc:359] Metadata directory not provided
I20260620 21:47:51.398052 13996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:47:51.398279 13996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:47:51 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992071, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:51.417963 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "1d4549a31dfd47bcab07c5ad66500287"
format_stamp: "Formatted at 2026-06-20 21:47:51 on dist-test-slave-5bvn"
server_key: "7d895965ea7bc996de482cf2d85ea796"
server_key_iv: "dc146a7237ce64b8d306ed9f1f43c3ac"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:51.418690 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "1d4549a31dfd47bcab07c5ad66500287"
format_stamp: "Formatted at 2026-06-20 21:47:51 on dist-test-slave-5bvn"
server_key: "7d895965ea7bc996de482cf2d85ea796"
server_key_iv: "dc146a7237ce64b8d306ed9f1f43c3ac"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:51.423672 13996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 21:47:51.442291 14013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:51.443945 13996 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260620 21:47:51.444082 13996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "1d4549a31dfd47bcab07c5ad66500287"
format_stamp: "Formatted at 2026-06-20 21:47:51 on dist-test-slave-5bvn"
server_key: "7d895965ea7bc996de482cf2d85ea796"
server_key_iv: "dc146a7237ce64b8d306ed9f1f43c3ac"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:51.444226 13996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:47:51.458221 13996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:47:51.461783 13996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:47:51.462000 13996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:47:51.462641 13996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:47:51.463625 13996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:47:51.463688 13996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:51.463752 13996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:47:51.463778 13996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:51.473412 13996 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:39457
I20260620 21:47:51.474538 13996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:47:51.478920 14126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:39457 every 8 connection(s)
I20260620 21:47:51.483217 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 13996
I20260620 21:47:51.483353 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:47:51.483662 1646 external_mini_cluster.cc:1428] Setting key 57a3734fc051e3bcf46206d8f2748dbc
Jun 20 21:47:51 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992071, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
I20260620 21:47:51.499291 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:51.483151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:37651 (local address 127.1.155.190:42873)
0620 21:47:51.483592 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:51.483602 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:51.485284 (+ 1682us) server_negotiation.cc:407] Connection header received
0620 21:47:51.485343 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:51.485346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:51.485404 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:51.485491 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:51.487953 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:51.488874 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:51.490724 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:51.491173 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:51.491267 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:51.494617 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:51.494622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:51.494627 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:51.494659 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:51.496694 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:51.497372 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:51.497377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:51.497382 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:51.497476 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:51.497935 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:51.497940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:51.497942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:51.498168 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:51.498252 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:51.498967 (+ 715us) server_negotiation.cc:299] Negotiation successful
0620 21:47:51.499101 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":127,"threads_started":1}
I20260620 21:47:51.504290 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:51.482041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42873 (local address 127.1.155.129:37651)
0620 21:47:51.483928 (+ 1887us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:51.483988 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:51.485000 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:51.485789 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:51.485799 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:51.486192 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:51.486881 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:51.486901 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:51.489051 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:51.489055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:51.490573 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:51.490587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:51.491510 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:51.491517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:51.491752 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:51.492450 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:51.492474 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:51.494403 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:51.496828 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:51.496834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:51.496851 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:51.497245 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:51.497605 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:51.497608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:51.497610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:51.497785 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:51.498319 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:51.498327 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:51.498666 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 21:47:51.499523 (+ 857us) client_negotiation.cc:240] Negotiation successful
0620 21:47:51.499885 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1605,"thread_start_us":132,"threads_started":1}
I20260620 21:47:51.505793 14127 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42873
I20260620 21:47:51.506098 14127 heartbeater.cc:464] Registering TS with master...
I20260620 21:47:51.506716 14127 heartbeater.cc:511] Master 127.1.155.190:42873 requested a full tablet report, sending...
I20260620 21:47:51.508622 13915 ts_manager.cc:194] Registered new tserver with Master: 1d4549a31dfd47bcab07c5ad66500287 (127.1.155.129:39457)
I20260620 21:47:51.510234 13915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:37651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:47:51.534078 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.155.148:47021/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42873
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:47:51 dist-test-slave-5bvn krb5kdc[13439](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992071, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 21:47:51.672490 14135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:47:51.672770 14135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:47:51.672873 14135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:47:51.677167 14135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:47:51.677249 14135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:47:51.677363 14135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:47:51.683411 14135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.155.148:47021/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:47:51.684677 14135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:47:51.685787 14135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:47:51.695856 14144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:51.696128 14142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:51.695858 14141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:51.697271 14135 server_base.cc:1034] running on GCE node
I20260620 21:47:51.697741 14135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:47:51.698480 14135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:47:51.699736 14135 hybrid_clock.cc:648] HybridClock initialized: now 1781992071699690 us; error 51 us; skew 500 ppm
Jun 20 21:47:51 dist-test-slave-5bvn krb5kdc[13439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992071, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:51.703248 14135 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:47:51.704609 14135 webserver.cc:460] Webserver started at http://127.1.155.130:36701/ using document root <none> and password file <none>
I20260620 21:47:51.705344 14135 fs_manager.cc:359] Metadata directory not provided
I20260620 21:47:51.705453 14135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:47:51.705777 14135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:47:51 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992071, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:51.725073 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "2a4d24df36284d599c9a687d070e25ec"
format_stamp: "Formatted at 2026-06-20 21:47:51 on dist-test-slave-5bvn"
server_key: "e3278769d9b2fa45059f3ecd36f6bfb0"
server_key_iv: "7b8eb9171b75a293653b1d6baa2f06e7"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:51.727242 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "2a4d24df36284d599c9a687d070e25ec"
format_stamp: "Formatted at 2026-06-20 21:47:51 on dist-test-slave-5bvn"
server_key: "e3278769d9b2fa45059f3ecd36f6bfb0"
server_key_iv: "7b8eb9171b75a293653b1d6baa2f06e7"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:51.735313 14135 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.004s
I20260620 21:47:51.756979 14151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:51.759203 14135 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 21:47:51.759372 14135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "2a4d24df36284d599c9a687d070e25ec"
format_stamp: "Formatted at 2026-06-20 21:47:51 on dist-test-slave-5bvn"
server_key: "e3278769d9b2fa45059f3ecd36f6bfb0"
server_key_iv: "7b8eb9171b75a293653b1d6baa2f06e7"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:51.759536 14135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:47:51.788023 14135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:47:51.791903 14135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:47:51.792085 14135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:47:51.792758 14135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:47:51.793762 14135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:47:51.793828 14135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:51.793913 14135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:47:51.793946 14135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:51.807672 14135 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:46453
I20260620 21:47:51.808823 14135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:47:51.810225 14264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:46453 every 8 connection(s)
I20260620 21:47:51.817457 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 14135
I20260620 21:47:51.817600 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:47:51.817886 1646 external_mini_cluster.cc:1428] Setting key c90dad43f398d06f2fb514e71cdc959a
Jun 20 21:47:51 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992071, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:47:51.834697 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:51.814794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:53233 (local address 127.1.155.190:42873)
0620 21:47:51.814952 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:51.814957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:51.815509 (+ 552us) server_negotiation.cc:407] Connection header received
0620 21:47:51.816751 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:51.816755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:51.816827 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:51.816883 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:51.818558 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:51.819430 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:51.821183 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:51.821546 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:51.821623 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:51.824738 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:51.824742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:51.824745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:51.824770 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:51.831820 (+ 7050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:51.832634 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:51.832639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:51.832641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:51.832741 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:51.833205 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:51.833209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:51.833211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:51.833457 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:51.833550 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:51.834351 (+ 801us) server_negotiation.cc:299] Negotiation successful
0620 21:47:51.834503 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:47:51.836848 14267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:51.813602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42873 (local address 127.1.155.130:53233)
0620 21:47:51.815337 (+ 1735us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:51.815398 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:51.816490 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:51.817296 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:51.817305 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:51.817676 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:51.818379 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:51.818398 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:51.819605 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:51.819608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:51.821042 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:51.821053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:51.821820 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:51.821824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:51.822077 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:51.822754 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:51.822775 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:51.824540 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:51.832045 (+ 7505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:51.832053 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:51.832074 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:51.832489 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:51.832865 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:51.832868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:51.832870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:51.833079 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:51.833625 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:51.833633 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:51.834025 (+ 392us) client_negotiation.cc:769] Sending connection context
0620 21:47:51.835382 (+ 1357us) client_negotiation.cc:240] Negotiation successful
0620 21:47:51.835761 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1492,"thread_start_us":170,"threads_started":1}
I20260620 21:47:51.839748 14265 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42873
I20260620 21:47:51.840009 14265 heartbeater.cc:464] Registering TS with master...
I20260620 21:47:51.840598 14265 heartbeater.cc:511] Master 127.1.155.190:42873 requested a full tablet report, sending...
I20260620 21:47:51.842646 13915 ts_manager.cc:194] Registered new tserver with Master: 2a4d24df36284d599c9a687d070e25ec (127.1.155.130:46453)
I20260620 21:47:51.843400 13915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:53233
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:47:51.878036 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.155.148:47021/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42873
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:47:52.058914 14270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:47:52.059271 14270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:47:52.059332 14270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:47:52.065522 14270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:47:52.065611 14270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:47:52.065696 14270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:47:52.073387 14270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.155.148:47021/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:42873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:47:52.074782 14270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:47:52.076052 14270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:47:52.088820 14278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:52.088788 14275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:52.089107 14276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:52.089298 14270 server_base.cc:1034] running on GCE node
I20260620 21:47:52.089922 14270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:47:52.090711 14270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Jun 20 21:47:52 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992071, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:52.122391 14270 hybrid_clock.cc:648] HybridClock initialized: now 1781992072122257 us; error 149 us; skew 500 ppm
Jun 20 21:47:52 dist-test-slave-5bvn krb5kdc[13439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:52.126394 14270 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:47:52.127624 14270 webserver.cc:460] Webserver started at http://127.1.155.131:40003/ using document root <none> and password file <none>
I20260620 21:47:52.128343 14270 fs_manager.cc:359] Metadata directory not provided
I20260620 21:47:52.128427 14270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:47:52.128647 14270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:47:52 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:52.189057 14270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "4aa2cb67e08344c2955d4670d11d7e9c"
format_stamp: "Formatted at 2026-06-20 21:47:52 on dist-test-slave-5bvn"
server_key: "eb5d2e61b428bee423c892fb7c3ce787"
server_key_iv: "037e398895efe3bc863463c5a8a2cde0"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:52.190636 14270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "4aa2cb67e08344c2955d4670d11d7e9c"
format_stamp: "Formatted at 2026-06-20 21:47:52 on dist-test-slave-5bvn"
server_key: "eb5d2e61b428bee423c892fb7c3ce787"
server_key_iv: "037e398895efe3bc863463c5a8a2cde0"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:52.211772 14270 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.004s sys 0.000s
I20260620 21:47:52.262315 14285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:52.264168 14270 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.001s sys 0.000s
I20260620 21:47:52.264307 14270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "4aa2cb67e08344c2955d4670d11d7e9c"
format_stamp: "Formatted at 2026-06-20 21:47:52 on dist-test-slave-5bvn"
server_key: "eb5d2e61b428bee423c892fb7c3ce787"
server_key_iv: "037e398895efe3bc863463c5a8a2cde0"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:52.264443 14270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:47:52.309659 14270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:47:52.313062 14270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:47:52.313261 14270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:47:52.313925 14270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:47:52.314877 14270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:47:52.314947 14270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:52.315013 14270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:47:52.315045 14270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:52.326133 14270 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:45707
I20260620 21:47:52.327229 14270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:47:52.329391 14398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:45707 every 8 connection(s)
I20260620 21:47:52.332935 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 14270
I20260620 21:47:52.333098 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:47:52.333437 1646 external_mini_cluster.cc:1428] Setting key c177044b9e0294ce09e2b8d15616cdad
Jun 20 21:47:52 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:47:52.351648 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:52.333084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:47093 (local address 127.1.155.190:42873)
0620 21:47:52.333334 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:52.333338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:52.335614 (+ 2276us) server_negotiation.cc:407] Connection header received
0620 21:47:52.336762 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:52.336765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:52.336814 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:52.336861 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:52.338457 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:52.339329 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:52.341402 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:52.341722 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:52.341806 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:52.344668 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:52.344672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:52.344675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:52.344700 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:52.348343 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:52.349083 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:52.349089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:52.349092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:52.349189 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:52.349754 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:52.349758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:52.349759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:52.350009 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:52.350094 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:52.350882 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 21:47:52.351419 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260620 21:47:52.353305 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:52.332066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42873 (local address 127.1.155.131:47093)
0620 21:47:52.334543 (+ 2477us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:52.335536 (+ 993us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:52.336521 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:52.337246 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:52.337255 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:52.337603 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:52.338284 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:52.338303 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:52.339965 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:52.339969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:52.341270 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:52.341281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:52.342029 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:52.342033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:52.342218 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:52.342881 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:52.342901 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:52.344484 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:52.348510 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:52.348516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:52.348538 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:52.348951 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:52.349402 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:52.349407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:52.349410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:52.349618 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:52.350163 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:52.350169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:52.350586 (+ 417us) client_negotiation.cc:769] Sending connection context
0620 21:47:52.351888 (+ 1302us) client_negotiation.cc:240] Negotiation successful
0620 21:47:52.352264 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1606,"thread_start_us":101,"threads_started":1}
I20260620 21:47:52.354568 14399 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42873
I20260620 21:47:52.354813 14399 heartbeater.cc:464] Registering TS with master...
I20260620 21:47:52.355356 14399 heartbeater.cc:511] Master 127.1.155.190:42873 requested a full tablet report, sending...
I20260620 21:47:52.356364 13915 ts_manager.cc:194] Registered new tserver with Master: 4aa2cb67e08344c2955d4670d11d7e9c (127.1.155.131:45707)
I20260620 21:47:52.357061 13915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:47093
I20260620 21:47:52.360361 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:47:52.361291 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 13996
I20260620 21:47:52.369375 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 14135
I20260620 21:47:52.380434 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 14270
I20260620 21:47:52.391494 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 13883
I20260620 21:47:52.401937 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:42873
--webserver_interface=127.1.155.190
--webserver_port=44617
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:42873
--ranger_config_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.155.148:47021/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.155.148:47021/kms with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:47:52.522432 14402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:47:52.522691 14402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:47:52.522765 14402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:47:52.526883 14402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:47:52.526947 14402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:47:52.526966 14402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:47:52.526983 14402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:47:52.527034 14402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:47:52.532475 14402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.155.148:47021/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:42873
--ranger_config_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:42873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:47:52.533527 14402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:47:52.534638 14402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:47:52.543284 14410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:52.543268 14407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:52.543273 14408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:52.543339 14402 server_base.cc:1034] running on GCE node
I20260620 21:47:52.544075 14402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:47:52.545185 14402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:47:52.546356 14402 hybrid_clock.cc:648] HybridClock initialized: now 1781992072546345 us; error 30 us; skew 500 ppm
Jun 20 21:47:52 dist-test-slave-5bvn krb5kdc[13439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:52.549057 14402 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:47:52.549983 14402 webserver.cc:460] Webserver started at http://127.1.155.190:44617/ using document root <none> and password file <none>
I20260620 21:47:52.550525 14402 fs_manager.cc:359] Metadata directory not provided
I20260620 21:47:52.550618 14402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 21:47:52 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:52.592664 14402 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:47:52.594940 14418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:52.596203 14402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:47:52.596369 14402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "862a830f680d408c8e165cd6bd561322"
format_stamp: "Formatted at 2026-06-20 21:47:50 on dist-test-slave-5bvn"
server_key: "cc7126dd040a562d7565513d48e9f4bc"
server_key_iv: "59fc0f196181dd7ca1e48613c17336aa"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:52.596911 14402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:47:52.614296 14402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:47:52.616911 14402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:47:52.617115 14402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:47:52.624281 14402 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:42873
I20260620 21:47:52.624291 14470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:42873 every 8 connection(s)
I20260620 21:47:52.625365 14402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:47:52.629429 14471 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:47:52.629468 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 14402
I20260620 21:47:52.629977 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.155.148:47021/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:39457
--local_ip_for_outbound_sockets=127.1.155.129
--tserver_master_addrs=127.1.155.190:42873
--webserver_port=38995
--webserver_interface=127.1.155.129
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.155.148:47021/kms with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:47:52.631163 14471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322: Bootstrap starting.
I20260620 21:47:52.647627 14471 log.cc:824] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322: Log is configured to *not* fsync() on all Append() calls
I20260620 21:47:52.654496 14471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:47:52.654902 14471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322: Bootstrap complete.
I20260620 21:47:52.659099 14471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a830f680d408c8e165cd6bd561322" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42873 } }
I20260620 21:47:52.659526 14471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862a830f680d408c8e165cd6bd561322, State: Initialized, Role: FOLLOWER
I20260620 21:47:52.660074 14471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a830f680d408c8e165cd6bd561322" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42873 } }
I20260620 21:47:52.660228 14471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:47:52.660317 14471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:47:52.660431 14471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:47:52.662412 14471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a830f680d408c8e165cd6bd561322" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42873 } }
I20260620 21:47:52.662743 14471 leader_election.cc:302] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 862a830f680d408c8e165cd6bd561322; no voters:
I20260620 21:47:52.663082 14471 leader_election.cc:288] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 21:47:52.663208 14475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:47:52.663455 14475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [term 2 LEADER]: Becoming Leader. State: Replica: 862a830f680d408c8e165cd6bd561322, State: Running, Role: LEADER
I20260620 21:47:52.663882 14475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a830f680d408c8e165cd6bd561322" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42873 } }
I20260620 21:47:52.664285 14471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:47:52.664851 14475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 862a830f680d408c8e165cd6bd561322. Latest consensus state: current_term: 2 leader_uuid: "862a830f680d408c8e165cd6bd561322" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a830f680d408c8e165cd6bd561322" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42873 } } }
I20260620 21:47:52.664968 14475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [sys.catalog]: This master's current role is: LEADER
I20260620 21:47:52.665581 14477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "862a830f680d408c8e165cd6bd561322" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a830f680d408c8e165cd6bd561322" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 42873 } } }
I20260620 21:47:52.665701 14477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322 [sys.catalog]: This master's current role is: LEADER
I20260620 21:47:52.666539 14480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:47:52.666826 14471 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 21:47:52.672344 14480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:47:52.673099 14480 catalog_manager.cc:1259] Loaded cluster ID: 3af3066a08ac4584bb92b607be5c5b71
I20260620 21:47:52.673161 14480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:47:52.676014 14480 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:47:52.676679 14480 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322: Loaded TSK: 0
I20260620 21:47:52.677878 14480 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:47:52.775938 14473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:47:52.776292 14473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:47:52.776357 14473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:47:52.782678 14473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:47:52.782770 14473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:47:52.782899 14473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:47:52.791075 14473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.155.148:47021/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:39457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=38995
--tserver_master_addrs=127.1.155.190:42873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:47:52.792424 14473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:47:52.793743 14473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:47:52.806658 14509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:52.809531 14511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:52.809540 14508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:52.810464 14473 server_base.cc:1034] running on GCE node
I20260620 21:47:52.810958 14473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:47:52.811810 14473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:47:52.813015 14473 hybrid_clock.cc:648] HybridClock initialized: now 1781992072812984 us; error 44 us; skew 500 ppm
Jun 20 21:47:52 dist-test-slave-5bvn krb5kdc[13439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:52.816651 14473 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:47:52.817828 14473 webserver.cc:460] Webserver started at http://127.1.155.129:38995/ using document root <none> and password file <none>
I20260620 21:47:52.818559 14473 fs_manager.cc:359] Metadata directory not provided
I20260620 21:47:52.818671 14473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 21:47:52 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:52.840839 14473 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:47:52.848999 14519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:52.850703 14473 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260620 21:47:52.850848 14473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "1d4549a31dfd47bcab07c5ad66500287"
format_stamp: "Formatted at 2026-06-20 21:47:51 on dist-test-slave-5bvn"
server_key: "7d895965ea7bc996de482cf2d85ea796"
server_key_iv: "dc146a7237ce64b8d306ed9f1f43c3ac"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:52.851549 14473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:47:52.872043 14473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:47:52.875903 14473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:47:52.876137 14473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:47:52.876855 14473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:47:52.878016 14473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:47:52.878083 14473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:52.878151 14473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:47:52.878185 14473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:52.889992 14473 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:39457
I20260620 21:47:52.891362 14473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:47:52.892290 14632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:39457 every 8 connection(s)
I20260620 21:47:52.899062 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 14473
I20260620 21:47:52.899572 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.155.148:47021/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:46453
--local_ip_for_outbound_sockets=127.1.155.130
--tserver_master_addrs=127.1.155.190:42873
--webserver_port=36701
--webserver_interface=127.1.155.130
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.155.148:47021/kms with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:47:52 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:47:52.915716 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:52.895122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42873 (local address 127.1.155.129:35853)
0620 21:47:52.895762 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:52.895820 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:52.897609 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:52.899542 (+ 1933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:52.899551 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:52.899932 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:52.900693 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:52.900716 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:52.902917 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:52.902921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:52.904458 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:52.904470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:52.905951 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:52.905956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:52.906145 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:52.906850 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:52.906872 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:52.908666 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:52.911628 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:52.911635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:52.911650 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:52.912002 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:52.912526 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:52.912529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:52.912531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:52.912709 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:52.913633 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:52.913642 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:52.913988 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 21:47:52.914207 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 21:47:52.914584 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":183,"threads_started":1}
I20260620 21:47:52.915897 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:52.896114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:35853 (local address 127.1.155.190:42873)
0620 21:47:52.896649 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:52.896673 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:52.896734 (+ 61us) server_negotiation.cc:407] Connection header received
0620 21:47:52.898159 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:52.898176 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:52.898618 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:52.899005 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:52.900910 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:52.902731 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:52.904603 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:52.905547 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:52.905706 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:52.908922 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:52.908929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:52.908947 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:52.908982 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:52.911465 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:52.912176 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:52.912187 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:52.912197 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:52.912377 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:52.912832 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:52.912838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:52.912844 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:52.913462 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:52.913621 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:52.914841 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0620 21:47:52.915136 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":113,"threads_started":1}
I20260620 21:47:52.919266 14633 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42873
I20260620 21:47:52.919499 14633 heartbeater.cc:464] Registering TS with master...
I20260620 21:47:52.920022 14633 heartbeater.cc:511] Master 127.1.155.190:42873 requested a full tablet report, sending...
W20260620 21:47:52.922245 14633 heartbeater.cc:644] Failed to heartbeat to 127.1.155.190:42873 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 21:47:52.922377 14633 heartbeater.cc:464] Registering TS with master...
I20260620 21:47:52.922570 14633 heartbeater.cc:511] Master 127.1.155.190:42873 requested a full tablet report, sending...
I20260620 21:47:52.923403 14633 heartbeater.cc:464] Registering TS with master...
I20260620 21:47:52.923933 14633 heartbeater.cc:511] Master 127.1.155.190:42873 requested a full tablet report, sending...
W20260620 21:47:52.924710 14633 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:47:53.053756 14637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:47:53.054143 14637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:47:53.054214 14637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:47:53.058735 14637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:47:53.058815 14637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:47:53.058917 14637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:47:53.064884 14637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.155.148:47021/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:46453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=36701
--tserver_master_addrs=127.1.155.190:42873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:47:53.066116 14637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:47:53.067148 14637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:47:53.077750 14643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:53.078660 14645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:53.079157 14642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:53.083221 14637 server_base.cc:1034] running on GCE node
I20260620 21:47:53.083727 14637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:47:53.084555 14637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:47:53.085753 14637 hybrid_clock.cc:648] HybridClock initialized: now 1781992073085712 us; error 52 us; skew 500 ppm
Jun 20 21:47:53 dist-test-slave-5bvn krb5kdc[13439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:53.089553 14637 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:47:53.090812 14637 webserver.cc:460] Webserver started at http://127.1.155.130:36701/ using document root <none> and password file <none>
I20260620 21:47:53.091486 14637 fs_manager.cc:359] Metadata directory not provided
I20260620 21:47:53.091614 14637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 21:47:53 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:53.109776 14637 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:47:53.118120 14653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:53.120445 14637 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260620 21:47:53.120615 14637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "2a4d24df36284d599c9a687d070e25ec"
format_stamp: "Formatted at 2026-06-20 21:47:51 on dist-test-slave-5bvn"
server_key: "e3278769d9b2fa45059f3ecd36f6bfb0"
server_key_iv: "7b8eb9171b75a293653b1d6baa2f06e7"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:53.121351 14637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:47:53.156538 14637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:47:53.160552 14637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:47:53.160748 14637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:47:53.161423 14637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:47:53.162410 14637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:47:53.162469 14637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:53.162526 14637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:47:53.162557 14637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:53.172816 14637 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:46453
I20260620 21:47:53.173039 14767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:46453 every 8 connection(s)
I20260620 21:47:53.174125 14637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:47:53.174968 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 14637
I20260620 21:47:53.175422 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.155.148:47021/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:45707
--local_ip_for_outbound_sockets=127.1.155.131
--tserver_master_addrs=127.1.155.190:42873
--webserver_port=40003
--webserver_interface=127.1.155.131
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.155.148:47021/kms with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:47:53 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:47:53.191615 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:53.176204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:46697 (local address 127.1.155.190:42873)
0620 21:47:53.176337 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:53.176341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:53.178260 (+ 1919us) server_negotiation.cc:407] Connection header received
0620 21:47:53.179134 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:53.179138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:53.179199 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:53.179308 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:53.180793 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:53.181663 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:53.183421 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:53.183779 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:53.183870 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:53.186921 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:53.186926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:53.186929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:53.186956 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:53.188728 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:53.189243 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:53.189247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:53.189249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:53.189331 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:53.190522 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:53.190525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:53.190527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:53.190713 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:53.190824 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:53.191302 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 21:47:53.191440 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:47:53.192659 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:53.176656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42873 (local address 127.1.155.130:46697)
0620 21:47:53.178107 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:53.178149 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:53.178916 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:53.179501 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:53.179512 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:53.179861 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:53.180556 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:53.180581 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:53.181817 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:53.181824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:53.183274 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:53.183286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:53.184061 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:53.184065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:53.184276 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:53.184968 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:53.184989 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:53.186730 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:53.188853 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:53.188860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:53.188874 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:53.189135 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:53.189429 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:53.189432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:53.189433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:53.190402 (+ 969us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:53.190822 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:53.190829 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:53.191065 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 21:47:53.191249 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 21:47:53.191550 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1259,"thread_start_us":144,"threads_started":1}
I20260620 21:47:53.193986 14768 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42873
I20260620 21:47:53.194227 14768 heartbeater.cc:464] Registering TS with master...
I20260620 21:47:53.194772 14768 heartbeater.cc:511] Master 127.1.155.190:42873 requested a full tablet report, sending...
W20260620 21:47:53.196353 14768 heartbeater.cc:644] Failed to heartbeat to 127.1.155.190:42873 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 21:47:53.196465 14768 heartbeater.cc:464] Registering TS with master...
I20260620 21:47:53.196610 14768 heartbeater.cc:511] Master 127.1.155.190:42873 requested a full tablet report, sending...
I20260620 21:47:53.197386 14768 heartbeater.cc:464] Registering TS with master...
I20260620 21:47:53.197531 14768 heartbeater.cc:511] Master 127.1.155.190:42873 requested a full tablet report, sending...
W20260620 21:47:53.198268 14768 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:47:53.341282 14770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:47:53.341624 14770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:47:53.341704 14770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:47:53.351483 14770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:47:53.351567 14770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:47:53.351673 14770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:47:53.359892 14770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.155.148:47021/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:45707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=40003
--tserver_master_addrs=127.1.155.190:42873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:47:53.361272 14770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:47:53.362630 14770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:47:53.375159 14779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:53.375177 14777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:53.375201 14776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:53.375733 14770 server_base.cc:1034] running on GCE node
I20260620 21:47:53.376194 14770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:47:53.376962 14770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:47:53.378141 14770 hybrid_clock.cc:648] HybridClock initialized: now 1781992073378118 us; error 32 us; skew 500 ppm
Jun 20 21:47:53 dist-test-slave-5bvn krb5kdc[13439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:53.381439 14770 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:47:53.382532 14770 webserver.cc:460] Webserver started at http://127.1.155.131:40003/ using document root <none> and password file <none>
I20260620 21:47:53.383247 14770 fs_manager.cc:359] Metadata directory not provided
I20260620 21:47:53.383351 14770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 21:47:53 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:53.401132 14770 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:47:53.404196 14787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:53.405884 14770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:47:53.406064 14770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "4aa2cb67e08344c2955d4670d11d7e9c"
format_stamp: "Formatted at 2026-06-20 21:47:52 on dist-test-slave-5bvn"
server_key: "eb5d2e61b428bee423c892fb7c3ce787"
server_key_iv: "037e398895efe3bc863463c5a8a2cde0"
server_key_version: "kuduclusterkey@0"
I20260620 21:47:53.406844 14770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:47:53.424942 14770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:47:53.428040 14770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:47:53.428259 14770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:47:53.428897 14770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:47:53.429735 14770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:47:53.429781 14770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:53.429848 14770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:47:53.429914 14770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:53.440585 14770 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:45707
I20260620 21:47:53.440613 14900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:45707 every 8 connection(s)
I20260620 21:47:53.441740 14770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:47:53.444880 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 14770
Jun 20 21:47:53 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:47:53.457937 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:53.444259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:45283 (local address 127.1.155.190:42873)
0620 21:47:53.444389 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:53.444393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:53.445397 (+ 1004us) server_negotiation.cc:407] Connection header received
0620 21:47:53.446602 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:53.446604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:53.446642 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:53.446697 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:53.448231 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:53.448917 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:53.450524 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:53.450883 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:53.450963 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:53.453423 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:53.453427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:53.453430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:53.453454 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:53.454888 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:53.455540 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:53.455545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:53.455549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:53.455629 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:53.456029 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:53.456035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:53.456040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:53.456579 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:53.456656 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:53.457610 (+ 954us) server_negotiation.cc:299] Negotiation successful
0620 21:47:53.457732 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:47:53.459496 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:53.445901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54470 (local address 127.1.155.190:42873)
0620 21:47:53.446485 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:53.446489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:53.446501 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:47:53.446545 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:53.446550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:53.446604 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:53.447615 (+ 1011us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:53.447722 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:53.448539 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:53.449736 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:53.450095 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:53.450194 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:53.455891 (+ 5697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:53.455895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:53.455897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:53.455919 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:53.457608 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:53.458121 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:53.458126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:53.458129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:53.458218 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:53.458552 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:53.458556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:53.458558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:53.458781 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:53.458985 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:53.459191 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 21:47:53.459315 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":89,"threads_started":1}
I20260620 21:47:53.459825 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:53.444746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:42873 (local address 127.1.155.131:45283)
0620 21:47:53.445246 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:53.445296 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:53.446401 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:53.447108 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:53.447118 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:53.447453 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:53.448061 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:53.448079 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:53.449062 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:53.449065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:53.450395 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:53.450404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:53.451144 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:53.451150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:53.451380 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:53.451935 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:53.451951 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:53.453262 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:53.455021 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:53.455027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:53.455042 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:53.455407 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:53.455748 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:53.455751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:53.455754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:53.455924 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:53.456990 (+ 1066us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:53.456999 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:53.457323 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 21:47:53.458387 (+ 1064us) client_negotiation.cc:240] Negotiation successful
0620 21:47:53.458677 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":143,"threads_started":1}
I20260620 21:47:53.460927 14901 heartbeater.cc:347] Connected to a master server at 127.1.155.190:42873
I20260620 21:47:53.461179 14901 heartbeater.cc:464] Registering TS with master...
I20260620 21:47:53.461717 14901 heartbeater.cc:511] Master 127.1.155.190:42873 requested a full tablet report, sending...
I20260620 21:47:53.463063 14435 ts_manager.cc:194] Registered new tserver with Master: 4aa2cb67e08344c2955d4670d11d7e9c (127.1.155.131:45707)
I20260620 21:47:53.464259 14435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:45283
Jun 20 21:47:53 dist-test-slave-5bvn krb5kdc[13439](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:53.924980 14633 heartbeater.cc:464] Registering TS with master...
I20260620 21:47:53.925307 14633 heartbeater.cc:511] Master 127.1.155.190:42873 requested a full tablet report, sending...
I20260620 21:47:53.926101 14435 ts_manager.cc:194] Registered new tserver with Master: 1d4549a31dfd47bcab07c5ad66500287 (127.1.155.129:39457)
I20260620 21:47:53.926944 14435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:35853
Jun 20 21:47:54 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:54.198470 14768 heartbeater.cc:464] Registering TS with master...
I20260620 21:47:54.198767 14768 heartbeater.cc:511] Master 127.1.155.190:42873 requested a full tablet report, sending...
I20260620 21:47:54.199530 14435 ts_manager.cc:194] Registered new tserver with Master: 2a4d24df36284d599c9a687d070e25ec (127.1.155.130:46453)
I20260620 21:47:54.200407 14435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:46697
I20260620 21:47:54.207399 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:47:54.219726 14922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:54.209038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54480 (local address 127.1.155.190:42873)
0620 21:47:54.209627 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:54.209631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:54.209887 (+ 256us) server_negotiation.cc:407] Connection header received
0620 21:47:54.210078 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:54.210081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:54.210131 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:54.210202 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:54.211134 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.212043 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.213272 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.213693 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.213795 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:54.215639 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:54.215643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:54.215646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:54.215671 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:54.217493 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:54.218015 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:54.218020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:54.218022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:54.218108 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:54.218410 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:54.218413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:54.218415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:54.218633 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:54.218764 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:54.219421 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 21:47:54.219547 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":68,"threads_started":1}
I20260620 21:47:54.224035 14435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:47:54.228497 14435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:47:54.248008 14929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:54.240170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:45707 (local address 127.0.0.1:41226)
0620 21:47:54.241757 (+ 1587us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:54.241769 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:54.241897 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:54.243357 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:54.243363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:54.243394 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:47:54.243488 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:54.243494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.245257 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.245262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:54.246436 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:54.246447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.247508 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.247511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:54.247623 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:54.247667 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 21:47:54.247748 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 21:47:54.247861 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1524,"thread_start_us":94,"threads_started":1}
I20260620 21:47:54.249050 14930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:54.240579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.1.155.131:45707)
0620 21:47:54.242947 (+ 2368us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:54.242953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:54.242972 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:47:54.243041 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:54.243046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:54.243207 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:54.243632 (+ 425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:47:54.243767 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.244788 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.246591 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.247374 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.248623 (+ 1249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:54.248771 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:54.248822 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 21:47:54.248915 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2265,"thread_start_us":90,"threads_started":1}
I20260620 21:47:54.250290 14931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:54.239782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:46453 (local address 127.0.0.1:35862)
0620 21:47:54.241349 (+ 1567us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:54.241411 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:54.241535 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:54.242520 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:54.242526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:54.242547 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:47:54.242670 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:54.242677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.244985 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.244989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:54.248265 (+ 3276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:54.248279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.249866 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.249869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:54.249965 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:54.249998 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 21:47:54.250068 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 21:47:54.250138 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1395,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260620 21:47:54.252228 14835 tablet_service.cc:1467] Processing CreateTablet for tablet 270d6bf1f785427987e2d880f9bf2ac7 (DEFAULT_TABLE table=test-table [id=3e93c27d072b4962916089f750fa6ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:47:54.253338 14835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270d6bf1f785427987e2d880f9bf2ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:47:54.253893 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:54.240644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:39457 (local address 127.0.0.1:44862)
0620 21:47:54.242071 (+ 1427us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:54.242083 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:54.242149 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:54.244937 (+ 2788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:54.244940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:54.244959 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:47:54.245930 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:54.245941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.250496 (+ 4555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.250500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:54.252352 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:54.252363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.253502 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.253506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:54.253597 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:54.253621 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:47:54.253667 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 21:47:54.253731 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"mutex_wait_us":855,"thread_start_us":87,"threads_started":1}
I20260620 21:47:54.254577 14932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:54.240787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.1.155.129:39457)
0620 21:47:54.242407 (+ 1620us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:54.242414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:54.242437 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:47:54.242507 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:54.242511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:54.242684 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:54.245074 (+ 2390us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:47:54.246087 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.248613 (+ 2526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.252511 (+ 3898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.253316 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.254158 (+ 842us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:54.254307 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:54.254359 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 21:47:54.254448 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1500,"thread_start_us":86,"threads_started":1}
I20260620 21:47:54.255053 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:54.240916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35862 (local address 127.1.155.130:46453)
0620 21:47:54.241892 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:54.241900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:54.241925 (+ 25us) server_negotiation.cc:407] Connection header received
0620 21:47:54.242011 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:54.242016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:54.242214 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:54.242356 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:47:54.242801 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.244823 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.248748 (+ 3925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.249562 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.249665 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:54.249786 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:54.254775 (+ 4989us) server_negotiation.cc:299] Negotiation successful
0620 21:47:54.254886 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":115,"threads_started":1}
I20260620 21:47:54.257445 14567 tablet_service.cc:1467] Processing CreateTablet for tablet 270d6bf1f785427987e2d880f9bf2ac7 (DEFAULT_TABLE table=test-table [id=3e93c27d072b4962916089f750fa6ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:47:54.257815 14689 tablet_service.cc:1467] Processing CreateTablet for tablet 270d6bf1f785427987e2d880f9bf2ac7 (DEFAULT_TABLE table=test-table [id=3e93c27d072b4962916089f750fa6ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:47:54.258549 14567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270d6bf1f785427987e2d880f9bf2ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:47:54.258759 14689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270d6bf1f785427987e2d880f9bf2ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:47:54.267071 14936 tablet_bootstrap.cc:492] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec: Bootstrap starting.
I20260620 21:47:54.268647 14934 tablet_bootstrap.cc:492] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c: Bootstrap starting.
I20260620 21:47:54.269402 14936 tablet_bootstrap.cc:654] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec: Neither blocks nor log segments found. Creating new log.
I20260620 21:47:54.270310 14936 log.cc:824] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec: Log is configured to *not* fsync() on all Append() calls
I20260620 21:47:54.270663 14934 tablet_bootstrap.cc:654] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c: Neither blocks nor log segments found. Creating new log.
I20260620 21:47:54.271402 14934 log.cc:824] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:47:54.272033 14935 tablet_bootstrap.cc:492] T 270d6bf1f785427987e2d880f9bf2ac7 P 1d4549a31dfd47bcab07c5ad66500287: Bootstrap starting.
I20260620 21:47:54.273535 14934 tablet_bootstrap.cc:492] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c: No bootstrap required, opened a new log
I20260620 21:47:54.273772 14934 ts_tablet_manager.cc:1397] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 21:47:54.273934 14935 tablet_bootstrap.cc:654] T 270d6bf1f785427987e2d880f9bf2ac7 P 1d4549a31dfd47bcab07c5ad66500287: Neither blocks nor log segments found. Creating new log.
I20260620 21:47:54.275795 14935 log.cc:824] T 270d6bf1f785427987e2d880f9bf2ac7 P 1d4549a31dfd47bcab07c5ad66500287: Log is configured to *not* fsync() on all Append() calls
I20260620 21:47:54.278201 14935 tablet_bootstrap.cc:492] T 270d6bf1f785427987e2d880f9bf2ac7 P 1d4549a31dfd47bcab07c5ad66500287: No bootstrap required, opened a new log
I20260620 21:47:54.278378 14935 ts_tablet_manager.cc:1397] T 270d6bf1f785427987e2d880f9bf2ac7 P 1d4549a31dfd47bcab07c5ad66500287: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260620 21:47:54.278960 14934 raft_consensus.cc:348] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4d24df36284d599c9a687d070e25ec" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 46453 } } peers { permanent_uuid: "1d4549a31dfd47bcab07c5ad66500287" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 39457 } } peers { permanent_uuid: "4aa2cb67e08344c2955d4670d11d7e9c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45707 } }
I20260620 21:47:54.279305 14934 raft_consensus.cc:374] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:47:54.279395 14934 raft_consensus.cc:729] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa2cb67e08344c2955d4670d11d7e9c, State: Initialized, Role: FOLLOWER
I20260620 21:47:54.279935 14934 consensus_queue.cc:260] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4d24df36284d599c9a687d070e25ec" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 46453 } } peers { permanent_uuid: "1d4549a31dfd47bcab07c5ad66500287" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 39457 } } peers { permanent_uuid: "4aa2cb67e08344c2955d4670d11d7e9c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45707 } }
I20260620 21:47:54.281031 14934 ts_tablet_manager.cc:1428] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 21:47:54.281409 14901 heartbeater.cc:503] Master 127.1.155.190:42873 was elected leader, sending a full tablet report...
I20260620 21:47:54.282032 14935 raft_consensus.cc:348] T 270d6bf1f785427987e2d880f9bf2ac7 P 1d4549a31dfd47bcab07c5ad66500287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4d24df36284d599c9a687d070e25ec" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 46453 } } peers { permanent_uuid: "1d4549a31dfd47bcab07c5ad66500287" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 39457 } } peers { permanent_uuid: "4aa2cb67e08344c2955d4670d11d7e9c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45707 } }
I20260620 21:47:54.282262 14935 raft_consensus.cc:374] T 270d6bf1f785427987e2d880f9bf2ac7 P 1d4549a31dfd47bcab07c5ad66500287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:47:54.282307 14935 raft_consensus.cc:729] T 270d6bf1f785427987e2d880f9bf2ac7 P 1d4549a31dfd47bcab07c5ad66500287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4549a31dfd47bcab07c5ad66500287, State: Initialized, Role: FOLLOWER
I20260620 21:47:54.282753 14935 consensus_queue.cc:260] T 270d6bf1f785427987e2d880f9bf2ac7 P 1d4549a31dfd47bcab07c5ad66500287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4d24df36284d599c9a687d070e25ec" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 46453 } } peers { permanent_uuid: "1d4549a31dfd47bcab07c5ad66500287" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 39457 } } peers { permanent_uuid: "4aa2cb67e08344c2955d4670d11d7e9c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45707 } }
I20260620 21:47:54.284516 14935 ts_tablet_manager.cc:1428] T 270d6bf1f785427987e2d880f9bf2ac7 P 1d4549a31dfd47bcab07c5ad66500287: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260620 21:47:54.284821 14633 heartbeater.cc:503] Master 127.1.155.190:42873 was elected leader, sending a full tablet report...
I20260620 21:47:54.290740 14936 tablet_bootstrap.cc:492] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec: No bootstrap required, opened a new log
I20260620 21:47:54.290958 14936 ts_tablet_manager.cc:1397] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20260620 21:47:54.295073 14936 raft_consensus.cc:348] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4d24df36284d599c9a687d070e25ec" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 46453 } } peers { permanent_uuid: "1d4549a31dfd47bcab07c5ad66500287" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 39457 } } peers { permanent_uuid: "4aa2cb67e08344c2955d4670d11d7e9c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45707 } }
I20260620 21:47:54.295357 14936 raft_consensus.cc:374] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:47:54.295413 14936 raft_consensus.cc:729] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a4d24df36284d599c9a687d070e25ec, State: Initialized, Role: FOLLOWER
I20260620 21:47:54.295866 14936 consensus_queue.cc:260] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4d24df36284d599c9a687d070e25ec" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 46453 } } peers { permanent_uuid: "1d4549a31dfd47bcab07c5ad66500287" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 39457 } } peers { permanent_uuid: "4aa2cb67e08344c2955d4670d11d7e9c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45707 } }
I20260620 21:47:54.296559 14768 heartbeater.cc:503] Master 127.1.155.190:42873 was elected leader, sending a full tablet report...
I20260620 21:47:54.296983 14936 ts_tablet_manager.cc:1428] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 21:47:54.394111 14634 tablet.cc:2369] T 270d6bf1f785427987e2d880f9bf2ac7 P 1d4549a31dfd47bcab07c5ad66500287: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:47:54.417344 14940 raft_consensus.cc:482] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:47:54.417567 14940 raft_consensus.cc:504] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4d24df36284d599c9a687d070e25ec" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 46453 } } peers { permanent_uuid: "1d4549a31dfd47bcab07c5ad66500287" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 39457 } } peers { permanent_uuid: "4aa2cb67e08344c2955d4670d11d7e9c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45707 } }
I20260620 21:47:54.418870 14940 leader_election.cc:288] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a4d24df36284d599c9a687d070e25ec (127.1.155.130:46453), 1d4549a31dfd47bcab07c5ad66500287 (127.1.155.129:39457)
I20260620 21:47:54.424721 14932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:54.419619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:57357 (local address 127.1.155.129:39457)
0620 21:47:54.419765 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:54.419770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:54.420471 (+ 701us) server_negotiation.cc:407] Connection header received
0620 21:47:54.420665 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:54.420668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:54.420713 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:54.420757 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:47:54.421191 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.421956 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.423503 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.424074 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.424160 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:54.424209 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:54.424521 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 21:47:54.424598 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:47:54.425163 14943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:54.419238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:39457 (local address 127.1.155.131:57357)
0620 21:47:54.420384 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:54.420399 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:54.420550 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:54.420883 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:54.420885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:54.420900 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:47:54.421068 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:54.421074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.422086 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.422089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:54.423364 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:54.423374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.424295 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.424298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:54.424392 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:54.424412 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 21:47:54.424912 (+ 500us) client_negotiation.cc:240] Negotiation successful
0620 21:47:54.424995 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":1,"spinlock_wait_cycles":32512,"thread_start_us":117,"threads_started":1}
I20260620 21:47:54.425621 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:54.420196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:37089 (local address 127.1.155.130:46453)
0620 21:47:54.420326 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:54.420332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:54.420890 (+ 558us) server_negotiation.cc:407] Connection header received
0620 21:47:54.421131 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:54.421134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:54.421182 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:54.421227 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:47:54.421614 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.422402 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.423918 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.424530 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.424625 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:54.424683 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:54.425426 (+ 743us) server_negotiation.cc:299] Negotiation successful
0620 21:47:54.425506 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 21:47:54.425799 14582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270d6bf1f785427987e2d880f9bf2ac7" candidate_uuid: "4aa2cb67e08344c2955d4670d11d7e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4549a31dfd47bcab07c5ad66500287" is_pre_election: true
I20260620 21:47:54.426072 14944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:54.419840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:46453 (local address 127.1.155.131:37089)
0620 21:47:54.420790 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:54.420805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:54.420980 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:54.421354 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:54.421356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:54.421371 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:47:54.421488 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:54.421494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.422546 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.422549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:54.423773 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:54.423783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.424967 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.424972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:54.425086 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:54.425110 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:47:54.425810 (+ 700us) client_negotiation.cc:240] Negotiation successful
0620 21:47:54.425900 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":34,"thread_start_us":91,"threads_started":1}
I20260620 21:47:54.426218 14582 raft_consensus.cc:2454] T 270d6bf1f785427987e2d880f9bf2ac7 P 1d4549a31dfd47bcab07c5ad66500287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa2cb67e08344c2955d4670d11d7e9c in term 0.
I20260620 21:47:54.426662 14722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270d6bf1f785427987e2d880f9bf2ac7" candidate_uuid: "4aa2cb67e08344c2955d4670d11d7e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a4d24df36284d599c9a687d070e25ec" is_pre_election: true
I20260620 21:47:54.426769 14790 leader_election.cc:302] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4549a31dfd47bcab07c5ad66500287, 4aa2cb67e08344c2955d4670d11d7e9c; no voters:
I20260620 21:47:54.426959 14722 raft_consensus.cc:2454] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa2cb67e08344c2955d4670d11d7e9c in term 0.
I20260620 21:47:54.427084 14940 raft_consensus.cc:2781] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:47:54.427162 14940 raft_consensus.cc:482] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:47:54.427211 14940 raft_consensus.cc:3037] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [term 0 FOLLOWER]: Advancing to term 1
W20260620 21:47:54.427330 14769 tablet.cc:2369] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:47:54.428542 14940 raft_consensus.cc:504] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4d24df36284d599c9a687d070e25ec" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 46453 } } peers { permanent_uuid: "1d4549a31dfd47bcab07c5ad66500287" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 39457 } } peers { permanent_uuid: "4aa2cb67e08344c2955d4670d11d7e9c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45707 } }
I20260620 21:47:54.428941 14940 leader_election.cc:288] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [CANDIDATE]: Term 1 election: Requested vote from peers 2a4d24df36284d599c9a687d070e25ec (127.1.155.130:46453), 1d4549a31dfd47bcab07c5ad66500287 (127.1.155.129:39457)
I20260620 21:47:54.429353 14722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270d6bf1f785427987e2d880f9bf2ac7" candidate_uuid: "4aa2cb67e08344c2955d4670d11d7e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a4d24df36284d599c9a687d070e25ec"
I20260620 21:47:54.429473 14722 raft_consensus.cc:3037] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:47:54.429877 14582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270d6bf1f785427987e2d880f9bf2ac7" candidate_uuid: "4aa2cb67e08344c2955d4670d11d7e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4549a31dfd47bcab07c5ad66500287"
I20260620 21:47:54.430009 14582 raft_consensus.cc:3037] T 270d6bf1f785427987e2d880f9bf2ac7 P 1d4549a31dfd47bcab07c5ad66500287 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:47:54.430677 14722 raft_consensus.cc:2454] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa2cb67e08344c2955d4670d11d7e9c in term 1.
I20260620 21:47:54.431216 14582 raft_consensus.cc:2454] T 270d6bf1f785427987e2d880f9bf2ac7 P 1d4549a31dfd47bcab07c5ad66500287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa2cb67e08344c2955d4670d11d7e9c in term 1.
I20260620 21:47:54.431452 14791 leader_election.cc:302] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a4d24df36284d599c9a687d070e25ec, 4aa2cb67e08344c2955d4670d11d7e9c; no voters:
I20260620 21:47:54.431663 14940 raft_consensus.cc:2781] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:47:54.431876 14940 raft_consensus.cc:686] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [term 1 LEADER]: Becoming Leader. State: Replica: 4aa2cb67e08344c2955d4670d11d7e9c, State: Running, Role: LEADER
I20260620 21:47:54.432257 14940 consensus_queue.cc:237] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4d24df36284d599c9a687d070e25ec" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 46453 } } peers { permanent_uuid: "1d4549a31dfd47bcab07c5ad66500287" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 39457 } } peers { permanent_uuid: "4aa2cb67e08344c2955d4670d11d7e9c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45707 } }
I20260620 21:47:54.435619 14432 catalog_manager.cc:5495] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aa2cb67e08344c2955d4670d11d7e9c (127.1.155.131). New cstate: current_term: 1 leader_uuid: "4aa2cb67e08344c2955d4670d11d7e9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4d24df36284d599c9a687d070e25ec" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 46453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d4549a31dfd47bcab07c5ad66500287" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 39457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa2cb67e08344c2955d4670d11d7e9c" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45707 } health_report { overall_health: HEALTHY } } }
W20260620 21:47:54.443423 14902 tablet.cc:2369] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 21:47:54 dist-test-slave-5bvn krb5kdc[13439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for HTTP/127.1.155.148@KRBTEST.COM
I20260620 21:47:54.488482 14930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:54.483283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.1.155.131:45707)
0620 21:47:54.483702 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:54.483707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:54.483720 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:47:54.483806 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:54.483809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:54.483874 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:54.483973 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:47:54.484272 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.485028 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.486266 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:54.486596 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:54.486701 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:54.487134 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:47:54.487713 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:47:54.487831 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:54.488272 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 21:47:54.488333 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332}
I20260620 21:47:54.500041 14582 raft_consensus.cc:1261] T 270d6bf1f785427987e2d880f9bf2ac7 P 1d4549a31dfd47bcab07c5ad66500287 [term 1 FOLLOWER]: Refusing update from remote peer 4aa2cb67e08344c2955d4670d11d7e9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:47:54.500041 14722 raft_consensus.cc:1261] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec [term 1 FOLLOWER]: Refusing update from remote peer 4aa2cb67e08344c2955d4670d11d7e9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:47:54.500865 14940 consensus_queue.cc:1035] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d4549a31dfd47bcab07c5ad66500287" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 39457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:47:54.501659 14940 consensus_queue.cc:1035] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a4d24df36284d599c9a687d070e25ec" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 46453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:47:54.516593 14949 mvcc.cc:205] Tried to move back new op lower bound from 7299039537126346752 to 7299039536877748224. Current Snapshot: MvccSnapshot[applied={T|T < 7299039537126346752}]
I20260620 21:47:54.523842 14950 mvcc.cc:205] Tried to move back new op lower bound from 7299039537126346752 to 7299039536877748224. Current Snapshot: MvccSnapshot[applied={T|T < 7299039537126346752}]
I20260620 21:47:54.528720 14952 mvcc.cc:205] Tried to move back new op lower bound from 7299039537126346752 to 7299039536877748224. Current Snapshot: MvccSnapshot[applied={T|T < 7299039537126346752 or (T in {7299039537126346752})}]
I20260620 21:47:54.531769 14432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:47:54.531947 14432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:47:54.534698 14432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 862a830f680d408c8e165cd6bd561322: Sending DeleteTablet for 3 replicas of tablet 270d6bf1f785427987e2d880f9bf2ac7
I20260620 21:47:54.536216 14689 tablet_service.cc:1514] Processing DeleteTablet for tablet 270d6bf1f785427987e2d880f9bf2ac7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:47:54 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:35862
I20260620 21:47:54.536748 14567 tablet_service.cc:1514] Processing DeleteTablet for tablet 270d6bf1f785427987e2d880f9bf2ac7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:47:54 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:44862
I20260620 21:47:54.537178 14835 tablet_service.cc:1514] Processing DeleteTablet for tablet 270d6bf1f785427987e2d880f9bf2ac7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:47:54 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:41226
I20260620 21:47:54.538086 14960 tablet_replica.cc:330] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec: stopping tablet replica
I20260620 21:47:54.538421 14960 raft_consensus.cc:2229] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:47:54.538550 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 14473
I20260620 21:47:54.538698 14960 raft_consensus.cc:2258] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:47:54.539198 14962 tablet_replica.cc:330] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c: stopping tablet replica
I20260620 21:47:54.539465 14962 raft_consensus.cc:2229] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:47:54.539846 14962 raft_consensus.cc:2258] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:47:54.541431 14962 ts_tablet_manager.cc:1905] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:47:54.544430 14962 ts_tablet_manager.cc:1918] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:47:54.544524 14962 log.cc:1196] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/270d6bf1f785427987e2d880f9bf2ac7
I20260620 21:47:54.544831 14962 ts_tablet_manager.cc:1939] T 270d6bf1f785427987e2d880f9bf2ac7 P 4aa2cb67e08344c2955d4670d11d7e9c: Deleting consensus metadata
I20260620 21:47:54.547143 14960 ts_tablet_manager.cc:1905] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:47:54.549757 14960 ts_tablet_manager.cc:1918] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:47:54.549896 14960 log.cc:1196] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/270d6bf1f785427987e2d880f9bf2ac7
I20260620 21:47:54.550235 14960 ts_tablet_manager.cc:1939] T 270d6bf1f785427987e2d880f9bf2ac7 P 2a4d24df36284d599c9a687d070e25ec: Deleting consensus metadata
I20260620 21:47:54.551535 14422 catalog_manager.cc:4841] TS 2a4d24df36284d599c9a687d070e25ec (127.1.155.130:46453): tablet 270d6bf1f785427987e2d880f9bf2ac7 (table test-table [id=3e93c27d072b4962916089f750fa6ba2]) successfully deleted
I20260620 21:47:54.552032 14419 catalog_manager.cc:4841] TS 4aa2cb67e08344c2955d4670d11d7e9c (127.1.155.131:45707): tablet 270d6bf1f785427987e2d880f9bf2ac7 (table test-table [id=3e93c27d072b4962916089f750fa6ba2]) successfully deleted
W20260620 21:47:54.552588 14421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.155.129:39457 (error 108)
I20260620 21:47:54.554836 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 14637
I20260620 21:47:54.562817 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:54.562434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:39457 (local address 127.0.0.1:52712)
0620 21:47:54.562642 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:54.562721 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:39457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260620 21:47:54.563138 14421 catalog_manager.cc:4584] TS 1d4549a31dfd47bcab07c5ad66500287 (127.1.155.129:39457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 270d6bf1f785427987e2d880f9bf2ac7: Network error: Client connection negotiation failed: client connection to 127.1.155.129:39457: connect: Connection refused (error 111)
I20260620 21:47:54.565294 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 14770
I20260620 21:47:54.574080 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 14402
I20260620 21:47:54.586481 1646 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 21:47:54.932017 1646 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 21:47:54.932149 1646 mini_ranger.cc:66] Stopping Ranger...
I20260620 21:47:55.277963 1646 mini_ranger.cc:68] Stopped Ranger
2026-06-20 21:47:55.278 UTC [13469] LOG: received smart shutdown request
2026-06-20 21:47:55.282 UTC [13469] LOG: background worker "logical replication launcher" (PID 13478) exited with exit code 1
2026-06-20 21:47:55.282 UTC [13473] LOG: shutting down
2026-06-20 21:47:55.448 UTC [13469] LOG: database system is shut down
2026-06-20T21:47:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47715 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:47:55 dist-test-slave-5bvn krb5kdc[14972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:47:55 dist-test-slave-5bvn krb5kdc[14972](info): set up 2 sockets
Jun 20 21:47:55 dist-test-slave-5bvn krb5kdc[14972](info): commencing operation
krb5kdc: starting...
W20260620 21:47:56.618150 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:47:56 dist-test-slave-5bvn krb5kdc[14972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:47:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:47:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:47:56.740775 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:45263
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:45263,127.1.155.189:38379,127.1.155.188:41171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:47:56.848902 14985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:47:56.849157 14985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:47:56.849197 14985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:47:56.853246 14985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:47:56.853312 14985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:47:56.853332 14985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:47:56.853348 14985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:47:56.853363 14985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:47:56.858599 14985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:45263,127.1.155.189:38379,127.1.155.188:41171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:45263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:47:56.859634 14985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:47:56.860637 14985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:47:56.869266 14990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:56.869273 14993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:56.869223 14991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:56.869597 14985 server_base.cc:1034] running on GCE node
I20260620 21:47:56.870246 14985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:47:56.871307 14985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:47:56.872502 14985 hybrid_clock.cc:648] HybridClock initialized: now 1781992076872472 us; error 46 us; skew 500 ppm
Jun 20 21:47:56 dist-test-slave-5bvn krb5kdc[14972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992076, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:56.875303 14985 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:47:56.876222 14985 webserver.cc:460] Webserver started at http://127.1.155.190:35955/ using document root <none> and password file <none>
I20260620 21:47:56.876771 14985 fs_manager.cc:359] Metadata directory not provided
I20260620 21:47:56.876863 14985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:47:56.877095 14985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:47:56.879129 14985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "76da812ce0f9409d9ded0e6059c6baca"
format_stamp: "Formatted at 2026-06-20 21:47:56 on dist-test-slave-5bvn"
server_key: "ba47d9a6111b97f927a365d424be2aa5"
server_key_iv: "b18b81ba960d3627e8005aff964b7ee7"
server_key_version: "encryptionkey@0"
I20260620 21:47:56.879674 14985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "76da812ce0f9409d9ded0e6059c6baca"
format_stamp: "Formatted at 2026-06-20 21:47:56 on dist-test-slave-5bvn"
server_key: "ba47d9a6111b97f927a365d424be2aa5"
server_key_iv: "b18b81ba960d3627e8005aff964b7ee7"
server_key_version: "encryptionkey@0"
I20260620 21:47:56.883307 14985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:47:56.885697 15000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:56.886926 14985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:47:56.887076 14985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "76da812ce0f9409d9ded0e6059c6baca"
format_stamp: "Formatted at 2026-06-20 21:47:56 on dist-test-slave-5bvn"
server_key: "ba47d9a6111b97f927a365d424be2aa5"
server_key_iv: "b18b81ba960d3627e8005aff964b7ee7"
server_key_version: "encryptionkey@0"
I20260620 21:47:56.887189 14985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:47:56.899807 14985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:47:56.903090 14985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:47:56.903298 14985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:47:56.910931 14985 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:45263
I20260620 21:47:56.910957 15052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:45263 every 8 connection(s)
I20260620 21:47:56.911845 14985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:47:56.914507 15053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:47:56.917121 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 14985
I20260620 21:47:56.917256 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:47:56.917567 1646 external_mini_cluster.cc:1428] Setting key 906df38c3b31bdd30d894ffe0e94008f
I20260620 21:47:56.918828 15053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.155.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.189@KRBTEST.COM" created.
Jun 20 21:47:56 dist-test-slave-5bvn krb5kdc[14972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992076, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:47:56.935205 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:56.922506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56320 (local address 127.1.155.190:45263)
0620 21:47:56.922801 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:56.922819 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:56.922863 (+ 44us) server_negotiation.cc:407] Connection header received
0620 21:47:56.923416 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:56.923420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:56.923596 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:56.923799 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:56.924644 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:56.925600 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:56.926925 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:56.927323 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:56.927564 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:56.930408 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:56.930413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:56.930423 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:56.930443 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:56.932063 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:56.932691 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:56.932696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:56.932697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:56.932804 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:56.933190 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:56.933192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:56.933193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:56.933588 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:56.933692 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:56.934190 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 21:47:56.934437 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":193,"spinlock_wait_cycles":4992,"thread_start_us":86,"threads_started":1}
I20260620 21:47:56.935210 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:56.921504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:45263 (local address 127.0.0.1:56320)
0620 21:47:56.922099 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:56.922144 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:56.923104 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:56.923793 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:56.923798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:56.924244 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:56.924487 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:56.924501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:56.925729 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:56.925733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:56.926775 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:56.926794 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:56.927476 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:56.927480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:56.927622 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:56.928412 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:56.928428 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:56.930250 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:56.932182 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:56.932193 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:56.932205 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:56.932536 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:56.932912 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:56.932917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:56.932920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:56.933077 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:56.933712 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:56.933720 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:56.933922 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 21:47:56.934116 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 21:47:56.934534 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":156,"thread_start_us":138,"threads_started":1}
I20260620 21:47:56.937513 15053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:47:56.938321 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:56.938026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38379 (local address 127.0.0.1:35490)
0620 21:47:56.938144 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:56.938221 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:38379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 21:47:56.938583 15002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.155.189:38379: connect: Connection refused (error 111)
I20260620 21:47:56.939394 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:56.939213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38379 (local address 127.0.0.1:35504)
0620 21:47:56.939301 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:56.939348 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:38379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 21:47:56.939586 15053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.189:38379: Network error: Client connection negotiation failed: client connection to 127.1.155.189:38379: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.189.keytab.
Entry for principal kudu/127.1.155.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.189.keytab.
I20260620 21:47:56.945710 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.189:38379
--webserver_interface=127.1.155.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.189.keytab
--principal=kudu/127.1.155.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:45263,127.1.155.189:38379,127.1.155.188:41171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:47:56.988879 15053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } attempt: 1
I20260620 21:47:56.990089 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:56.989566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38379 (local address 127.0.0.1:35518)
0620 21:47:56.989917 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:56.990001 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:38379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 21:47:56.990963 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:56.990681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38379 (local address 127.0.0.1:35528)
0620 21:47:56.990839 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:56.990897 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:38379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 21:47:56.991129 15053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.189:38379: Network error: Client connection negotiation failed: client connection to 127.1.155.189:38379: connect: Connection refused (error 111)
W20260620 21:47:57.056298 15060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:47:57.056598 15060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:47:57.056641 15060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 21:47:57.059341 15053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } attempt: 2
I20260620 21:47:57.060181 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.059878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38379 (local address 127.0.0.1:35540)
0620 21:47:57.060034 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.060104 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:38379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 21:47:57.061008 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.060737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38379 (local address 127.0.0.1:35542)
0620 21:47:57.060875 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.060941 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:38379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 21:47:57.061234 15060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:47:57.061304 15060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:47:57.061317 15053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.189:38379: Network error: Client connection negotiation failed: client connection to 127.1.155.189:38379: connect: Connection refused (error 111)
W20260620 21:47:57.061398 15060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:47:57.061429 15060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:47:57.061445 15060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:47:57.066953 15060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:45263,127.1.155.189:38379,127.1.155.188:41171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.189.keytab
--principal=kudu/127.1.155.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.189:38379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.155.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:47:57.067983 15060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:47:57.068902 15060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:47:57.077181 15065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:57.077236 15066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:57.077245 15060 server_base.cc:1034] running on GCE node
W20260620 21:47:57.077167 15068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:57.078006 15060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:47:57.079069 15060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:47:57.080226 15060 hybrid_clock.cc:648] HybridClock initialized: now 1781992077080195 us; error 48 us; skew 500 ppm
Jun 20 21:47:57 dist-test-slave-5bvn krb5kdc[14972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992077, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:57.083014 15060 init.cc:377] Logged in from keytab as kudu/127.1.155.189@KRBTEST.COM (short username kudu)
I20260620 21:47:57.083881 15060 webserver.cc:460] Webserver started at http://127.1.155.189:43357/ using document root <none> and password file <none>
I20260620 21:47:57.084376 15060 fs_manager.cc:359] Metadata directory not provided
I20260620 21:47:57.084432 15060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:47:57.084590 15060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:47:57.086586 15060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/data/instance:
uuid: "7603ab9512a1497c894eca5b9c4423ec"
format_stamp: "Formatted at 2026-06-20 21:47:57 on dist-test-slave-5bvn"
server_key: "bef3ae6c41f126424dd87e55cc345456"
server_key_iv: "db8791dcc6b993815f5b8391da1e22b5"
server_key_version: "encryptionkey@0"
I20260620 21:47:57.087082 15060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/wal/instance:
uuid: "7603ab9512a1497c894eca5b9c4423ec"
format_stamp: "Formatted at 2026-06-20 21:47:57 on dist-test-slave-5bvn"
server_key: "bef3ae6c41f126424dd87e55cc345456"
server_key_iv: "db8791dcc6b993815f5b8391da1e22b5"
server_key_version: "encryptionkey@0"
I20260620 21:47:57.090634 15060 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:47:57.093433 15075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:57.094619 15060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:47:57.094753 15060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/wal
uuid: "7603ab9512a1497c894eca5b9c4423ec"
format_stamp: "Formatted at 2026-06-20 21:47:57 on dist-test-slave-5bvn"
server_key: "bef3ae6c41f126424dd87e55cc345456"
server_key_iv: "db8791dcc6b993815f5b8391da1e22b5"
server_key_version: "encryptionkey@0"
I20260620 21:47:57.094859 15060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:47:57.104729 15060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:47:57.107985 15060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:47:57.108192 15060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:47:57.116799 15060 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.189:38379
I20260620 21:47:57.116809 15127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.189:38379 every 8 connection(s)
I20260620 21:47:57.117970 15060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/data/info.pb
I20260620 21:47:57.121265 15128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:47:57.122006 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 15060
I20260620 21:47:57.122138 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-1/wal/instance
I20260620 21:47:57.122409 1646 external_mini_cluster.cc:1428] Setting key 94d984466bdb0c6867f2547fe61e7e7c
I20260620 21:47:57.125396 15128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.155.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.188@KRBTEST.COM" created.
Jun 20 21:47:57 dist-test-slave-5bvn krb5kdc[14972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992077, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.189@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:47:57.141274 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.127524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.1.155.190:45263)
0620 21:47:57.127673 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:57.127677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:57.128899 (+ 1222us) server_negotiation.cc:407] Connection header received
0620 21:47:57.129909 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:57.129913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:57.130024 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:57.130139 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:57.131336 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.131906 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.133056 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.133342 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.133464 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.136202 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:57.136209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:57.136215 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:57.136250 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:57.138874 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.139472 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.139477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.139479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.139573 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.139922 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.139928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.139932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.140200 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:57.140317 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:57.140877 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 21:47:57.141024 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:47:57.142195 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.128108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:45263 (local address 127.0.0.1:56334)
0620 21:47:57.128714 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.128761 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:57.129666 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:57.130335 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:57.130345 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:57.130696 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:57.131182 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.131195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.132048 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.132054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.132927 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.132935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.133511 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.133515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.133696 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.134366 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:57.134382 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:57.136014 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:57.139015 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.139020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.139036 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.139335 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.139672 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.139674 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.139676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.139820 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.140308 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:57.140314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:57.140606 (+ 292us) client_negotiation.cc:769] Sending connection context
0620 21:47:57.140768 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 21:47:57.141053 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":165,"threads_started":1}
I20260620 21:47:57.143770 15128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:47:57 dist-test-slave-5bvn krb5kdc[14972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992077, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.189@KRBTEST.COM for kudu/127.1.155.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.188.keytab.
Entry for principal kudu/127.1.155.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.188.keytab.
I20260620 21:47:57.150888 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.188:41171
--webserver_interface=127.1.155.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.188.keytab
--principal=kudu/127.1.155.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:45263,127.1.155.189:38379,127.1.155.188:41171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:47:57.152482 15053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } attempt: 3
I20260620 21:47:57.154256 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.144294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38379 (local address 127.0.0.1:35550)
0620 21:47:57.144445 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.144457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:57.144524 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:57.145083 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:57.145086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:57.145300 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:57.145411 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.145417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.146666 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.146668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.147335 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.147344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.147986 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.147989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.148103 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.148604 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:57.148618 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:57.150027 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:57.151925 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.151930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.151935 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.152216 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.152568 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.152571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.152573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.152667 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.153632 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:57.153638 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:57.153737 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 21:47:57.153819 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 21:47:57.154000 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 21:47:57.155135 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.144398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.1.155.189:38379)
0620 21:47:57.144759 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:57.144767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:57.144784 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:47:57.144814 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:57.144818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:57.144957 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:57.145066 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:57.145542 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.146544 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.147489 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.147871 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.147969 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.150177 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:57.150186 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:57.150197 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:57.150217 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:57.151808 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.152329 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.152338 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.152341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.152467 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.153153 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.153158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.153162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.153516 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:57.154579 (+ 1063us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:57.154691 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 21:47:57.154853 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":176,"threads_started":1}
I20260620 21:47:57.158044 15128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:47:57 dist-test-slave-5bvn krb5kdc[14972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992076, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for kudu/127.1.155.189@KRBTEST.COM
I20260620 21:47:57.158910 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.158571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:41171 (local address 127.0.0.1:40796)
0620 21:47:57.158729 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.158820 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 21:47:57.159196 15079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
I20260620 21:47:57.160389 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.160131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:41171 (local address 127.0.0.1:40812)
0620 21:47:57.160250 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.160312 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 21:47:57.160601 15128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:41171: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
I20260620 21:47:57.163290 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.152980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38379 (local address 127.0.0.1:35554)
0620 21:47:57.153142 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.153161 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:57.153250 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:57.153580 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:57.153583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:57.153749 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:57.153907 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.153913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.154801 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.154804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.155570 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.155579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.156829 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.156832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.156955 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.157465 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:57.157477 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:57.158962 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:57.161600 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.161603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.161605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.161908 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.162275 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.162278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.162280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.162372 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.162878 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:57.162881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:57.162983 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 21:47:57.163067 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 21:47:57.163181 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 21:47:57.163434 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.153047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.1.155.189:38379)
0620 21:47:57.153375 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:57.153383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:57.153392 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:47:57.153418 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:57.153420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:57.153451 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:57.153562 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:57.154024 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.154690 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.155702 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.156713 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.156816 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.159124 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:57.159131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:57.159137 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:57.159169 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:57.161450 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.162025 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.162032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.162036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.162136 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.162493 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.162499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.162502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.162767 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:57.162884 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:57.163160 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 21:47:57.163275 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":97,"threads_started":1}
I20260620 21:47:57.164088 15053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:47:57.164759 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.164551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:41171 (local address 127.0.0.1:40822)
0620 21:47:57.164659 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.164704 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 21:47:57.165385 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.165169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:41171 (local address 127.0.0.1:40832)
0620 21:47:57.165299 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.165339 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 21:47:57.165570 15053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:41171: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
I20260620 21:47:57.196738 15053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } attempt: 1
I20260620 21:47:57.197654 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.197323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:41171 (local address 127.0.0.1:40848)
0620 21:47:57.197504 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.197570 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 21:47:57.198616 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.198373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:41171 (local address 127.0.0.1:40852)
0620 21:47:57.198514 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.198560 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 21:47:57.198864 15053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:41171: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
I20260620 21:47:57.210000 15128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } attempt: 1
I20260620 21:47:57.210887 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.210549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:41171 (local address 127.0.0.1:40858)
0620 21:47:57.210722 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.210795 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 21:47:57.211692 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.211459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:41171 (local address 127.0.0.1:40866)
0620 21:47:57.211573 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.211627 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 21:47:57.211869 15128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:41171: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
W20260620 21:47:57.272755 15134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:47:57.273037 15134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:47:57.273131 15134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 21:47:57.274063 15053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } attempt: 2
I20260620 21:47:57.274955 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.274652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:41171 (local address 127.0.0.1:40868)
0620 21:47:57.274811 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.274874 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 21:47:57.275700 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.275478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:41171 (local address 127.0.0.1:40874)
0620 21:47:57.275592 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.275645 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 21:47:57.275833 15053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:41171: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
W20260620 21:47:57.278138 15134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:47:57.278228 15134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:47:57.278275 15134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:47:57.278318 15134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:47:57.278373 15134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:47:57.280081 15128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } attempt: 2
I20260620 21:47:57.280861 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.280550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:41171 (local address 127.0.0.1:40886)
0620 21:47:57.280707 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.280774 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 21:47:57.281594 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.281357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:41171 (local address 127.0.0.1:40894)
0620 21:47:57.281482 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.281532 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 21:47:57.281730 15128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:41171: Network error: Client connection negotiation failed: client connection to 127.1.155.188:41171: connect: Connection refused (error 111)
I20260620 21:47:57.284699 15134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:45263,127.1.155.189:38379,127.1.155.188:41171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.188.keytab
--principal=kudu/127.1.155.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.188:41171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.155.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:47:57.285801 15134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:47:57.286885 15134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:47:57.295714 15144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:47:57.295955 15141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:57.295859 15134 server_base.cc:1034] running on GCE node
W20260620 21:47:57.295814 15142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:47:57.296727 15134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:47:57.298380 15134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:47:57.299612 15134 hybrid_clock.cc:648] HybridClock initialized: now 1781992077299569 us; error 61 us; skew 500 ppm
Jun 20 21:47:57 dist-test-slave-5bvn krb5kdc[14972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992077, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:47:57.302757 15134 init.cc:377] Logged in from keytab as kudu/127.1.155.188@KRBTEST.COM (short username kudu)
I20260620 21:47:57.303741 15134 webserver.cc:460] Webserver started at http://127.1.155.188:37347/ using document root <none> and password file <none>
I20260620 21:47:57.304335 15134 fs_manager.cc:359] Metadata directory not provided
I20260620 21:47:57.304430 15134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:47:57.304677 15134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:47:57.306900 15134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/data/instance:
uuid: "65fdad74142442d68800125606265d2b"
format_stamp: "Formatted at 2026-06-20 21:47:57 on dist-test-slave-5bvn"
server_key: "bf6e8266eaf11757619e93ef1ef6aca4"
server_key_iv: "2773698627c3ade9e5df09e7b3fe51ea"
server_key_version: "encryptionkey@0"
I20260620 21:47:57.307480 15134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/wal/instance:
uuid: "65fdad74142442d68800125606265d2b"
format_stamp: "Formatted at 2026-06-20 21:47:57 on dist-test-slave-5bvn"
server_key: "bf6e8266eaf11757619e93ef1ef6aca4"
server_key_iv: "2773698627c3ade9e5df09e7b3fe51ea"
server_key_version: "encryptionkey@0"
I20260620 21:47:57.311334 15134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:47:57.313772 15151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:47:57.315058 15134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:47:57.315205 15134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/wal
uuid: "65fdad74142442d68800125606265d2b"
format_stamp: "Formatted at 2026-06-20 21:47:57 on dist-test-slave-5bvn"
server_key: "bf6e8266eaf11757619e93ef1ef6aca4"
server_key_iv: "2773698627c3ade9e5df09e7b3fe51ea"
server_key_version: "encryptionkey@0"
I20260620 21:47:57.315330 15134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:47:57.330219 15134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:47:57.333015 15134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:47:57.333220 15134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:47:57.340309 15134 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.188:41171
I20260620 21:47:57.340338 15203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.188:41171 every 8 connection(s)
I20260620 21:47:57.341233 15134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/data/info.pb
I20260620 21:47:57.343920 15204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:47:57.347627 15204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:47:57.348409 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 15134
I20260620 21:47:57.348539 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781991920067886-1646-0/minicluster-data/master-2/wal/instance
I20260620 21:47:57.348805 1646 external_mini_cluster.cc:1428] Setting key 9544a84cc0db3d7d4bb4b9c534dc868e
Jun 20 21:47:57 dist-test-slave-5bvn krb5kdc[14972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992077, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.188@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
Jun 20 21:47:57 dist-test-slave-5bvn krb5kdc[14972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.188@KRBTEST.COM
I20260620 21:47:57.361716 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.349687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.1.155.190:45263)
0620 21:47:57.349894 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:57.349899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:57.350832 (+ 933us) server_negotiation.cc:407] Connection header received
0620 21:47:57.351646 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:57.351648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:57.351695 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:57.351768 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:57.352705 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.353201 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.354347 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.354658 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.354742 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.357527 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:57.357530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:57.357532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:57.357558 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:57.359343 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.359943 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.359948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.359949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.360047 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.360464 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.360468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.360470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.360739 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:57.360831 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:57.361393 (+ 562us) server_negotiation.cc:299] Negotiation successful
0620 21:47:57.361547 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 21:47:57.362541 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.350080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40896 (local address 127.1.155.188:41171)
0620 21:47:57.350620 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:57.350637 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:57.350674 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:47:57.351429 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:57.351451 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:57.351684 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:57.351877 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:57.352586 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.353554 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.354507 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.354931 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.355027 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.358023 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:57.358031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:57.358065 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:57.358094 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:57.359943 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.360522 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.360531 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.360533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.360616 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.360918 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.360920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.360922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.361167 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:57.361350 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:57.361591 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 21:47:57.361722 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":425,"spinlock_wait_cycles":8576,"thread_start_us":136,"threads_started":1}
I20260620 21:47:57.362562 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.350080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:45263 (local address 127.0.0.1:56344)
0620 21:47:57.350672 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.350722 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:57.351429 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:57.351839 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:57.351844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:57.352267 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:57.352549 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.352568 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.353308 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.353312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.354228 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.354236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.354797 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.354802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.354991 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.355762 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:57.355777 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:57.357305 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:57.359469 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.359477 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.359486 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.359832 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.360170 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.360173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.360175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.360346 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.360857 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:57.360867 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:57.361124 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 21:47:57.361305 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 21:47:57.361643 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":61,"spinlock_wait_cycles":46848,"thread_start_us":149,"threads_started":1}
I20260620 21:47:57.363819 15204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:47:57 dist-test-slave-5bvn krb5kdc[14972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992077, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.188@KRBTEST.COM for kudu/127.1.155.189@KRBTEST.COM
Jun 20 21:47:57 dist-test-slave-5bvn krb5kdc[14972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:47:57.372871 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.364468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.1.155.189:38379)
0620 21:47:57.364595 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:57.364599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:57.364615 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:47:57.364668 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:57.364670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:57.364709 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:57.364804 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:57.365247 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.365783 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.366613 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.366866 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.366980 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.369289 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:57.369294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:57.369296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:57.369323 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:57.371114 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.371596 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.371598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.371600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.371684 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.371986 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.371989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.372015 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.372237 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:57.372339 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:57.372610 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 21:47:57.372756 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:47:57.372874 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.364294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:38379 (local address 127.0.0.1:35570)
0620 21:47:57.364421 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.364432 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:57.364523 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:57.364818 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:57.364821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:57.365017 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:57.365139 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.365148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.365942 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.365953 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.366494 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.366501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.366982 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.366986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.367082 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.367582 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:57.367595 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:57.369147 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:57.371236 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.371242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.371248 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.371493 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.371776 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.371782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.371786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.371878 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.372333 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:57.372336 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:57.372434 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 21:47:57.372541 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 21:47:57.372671 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 21:47:57.372937 15128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } attempt: 3
I20260620 21:47:57.373770 15204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:47:57.374236 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.365019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.1.155.190:45263)
0620 21:47:57.365135 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:57.365139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:57.365153 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:47:57.365271 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:57.365274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:57.365321 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:57.365403 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:57.366033 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.366505 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.367264 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.367481 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.367604 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.370249 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:57.370253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:57.370255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:57.370282 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:57.371884 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.372333 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.372336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.372338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.372420 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.372731 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.372734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.372736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.372967 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:57.373227 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:57.373931 (+ 704us) server_negotiation.cc:299] Negotiation successful
0620 21:47:57.374074 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:47:57.374981 15053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } attempt: 3
Jun 20 21:47:57 dist-test-slave-5bvn krb5kdc[14972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992077, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.189@KRBTEST.COM for kudu/127.1.155.188@KRBTEST.COM
Jun 20 21:47:57 dist-test-slave-5bvn krb5kdc[14972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992077, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.188@KRBTEST.COM for kudu/127.1.155.188@KRBTEST.COM
Jun 20 21:47:57 dist-test-slave-5bvn krb5kdc[14972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992076, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for kudu/127.1.155.188@KRBTEST.COM
Jun 20 21:47:57 dist-test-slave-5bvn krb5kdc[14972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.189@KRBTEST.COM
I20260620 21:47:57.384014 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.373336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:41171 (local address 127.0.0.1:40910)
0620 21:47:57.373596 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.373614 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:57.373686 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:57.374094 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:57.374097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:57.374257 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:57.374358 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.374366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.375537 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.375541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.376445 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.376455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.377101 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.377105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.377191 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.377652 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:57.377664 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:57.380076 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:57.382197 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.382201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.382203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.382446 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.382772 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.382777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.382779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.382871 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.383438 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:57.383444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:57.383558 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 21:47:57.383651 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 21:47:57.383799 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260620 21:47:57.384800 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.373586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.1.155.188:41171)
0620 21:47:57.373870 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:57.373876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:57.373893 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:47:57.373928 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:57.373931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:57.373970 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:57.374057 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:57.374713 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.375422 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.376592 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.376977 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.377390 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.380248 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:57.380253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:57.380258 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:57.380299 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:57.382041 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.382558 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.382564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.382567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.382652 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.382999 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.383005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.383010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.383318 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:57.384341 (+ 1023us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:57.384423 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 21:47:57.384608 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":203,"spinlock_wait_cycles":7424}
I20260620 21:47:57.386029 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.374310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:41171 (local address 127.0.0.1:40924)
0620 21:47:57.374535 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.374558 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:57.374621 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:57.374926 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:57.374929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:57.375296 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:57.375372 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.375378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.376744 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.376746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.377536 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.377544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.378877 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.378883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.378960 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.379512 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:57.379526 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:57.380964 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:57.383556 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.383559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.383561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.383873 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.384387 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.384390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.384392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.384497 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.385317 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:57.385319 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:57.385405 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 21:47:57.385687 (+ 282us) client_negotiation.cc:240] Negotiation successful
0620 21:47:57.385825 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":17}
I20260620 21:47:57.386478 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.374333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.1.155.188:41171)
0620 21:47:57.374690 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:57.374700 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:57.374710 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:47:57.374734 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:57.374737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:57.374774 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:57.374847 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:57.375488 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.376654 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.378297 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.378682 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.378776 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.381099 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:57.381102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:57.381106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:57.381131 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:57.383445 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.383970 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.383973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.383974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.384050 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.384932 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.384935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.384936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.385183 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:57.385281 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:57.386244 (+ 963us) server_negotiation.cc:299] Negotiation successful
0620 21:47:57.386354 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":122,"threads_started":1}
I20260620 21:47:57.387979 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.375776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.1.155.189:38379)
0620 21:47:57.375884 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:57.375888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:57.375902 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:47:57.375932 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:57.375935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:57.375978 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:57.376085 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:57.376824 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.377945 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.378886 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.379121 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.379245 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.383898 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:57.383901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:57.383904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:57.383928 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:57.385800 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.386702 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.386705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.386707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.386788 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.387059 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.387063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.387065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.387293 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:57.387475 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:57.387682 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 21:47:57.387802 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 21:47:57.388152 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.375323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:41171 (local address 127.0.0.1:40934)
0620 21:47:57.375476 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 21:47:57.375491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:47:57.375573 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:47:57.376564 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:47:57.376567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:47:57.376726 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:47:57.376830 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.376836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.377981 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.377985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.378815 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:47:57.378825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.379925 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.379930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:47:57.380033 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.380438 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:47:57.380455 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:47:57.382033 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:47:57.385951 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.385955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.385957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.386211 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.387101 (+ 890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:47:57.387107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:47:57.387112 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:47:57.387200 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:47:57.387692 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:47:57.387695 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:47:57.387793 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 21:47:57.387876 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 21:47:57.388007 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 21:47:57.388686 15128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec: Bootstrap starting.
I20260620 21:47:57.388906 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.375756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.1.155.188:41171)
0620 21:47:57.376298 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:57.376306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:57.376321 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:47:57.376355 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:57.376358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:57.376414 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:57.376929 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:57.377063 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.377887 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.379355 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.379778 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.380252 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.382300 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:57.382307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:57.382311 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:57.382340 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:57.384794 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.386899 (+ 2105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.386902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.386903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.386979 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.387311 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.387314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.387316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.387573 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:57.388552 (+ 979us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:57.388648 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 21:47:57.388759 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"spinlock_wait_cycles":1792,"thread_start_us":95,"threads_started":1}
I20260620 21:47:57.391981 15128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec: Neither blocks nor log segments found. Creating new log.
I20260620 21:47:57.392433 15053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca: Bootstrap starting.
I20260620 21:47:57.392881 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:47:57.392953 15128 log.cc:824] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec: Log is configured to *not* fsync() on all Append() calls
I20260620 21:47:57.395252 15053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca: Neither blocks nor log segments found. Creating new log.
I20260620 21:47:57.395448 15128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec: No bootstrap required, opened a new log
I20260620 21:47:57.396116 15053 log.cc:824] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca: Log is configured to *not* fsync() on all Append() calls
I20260620 21:47:57.398290 15204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b: Bootstrap starting.
I20260620 21:47:57.398919 15128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } }
I20260620 21:47:57.399187 15128 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:47:57.399250 15128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7603ab9512a1497c894eca5b9c4423ec, State: Initialized, Role: FOLLOWER
I20260620 21:47:57.399698 15128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } }
I20260620 21:47:57.400496 15220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } } }
I20260620 21:47:57.400708 15220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:47:57.401046 15128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:47:57.401222 15204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b: Neither blocks nor log segments found. Creating new log.
I20260620 21:47:57.402140 15204 log.cc:824] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:47:57.404479 15053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca: No bootstrap required, opened a new log
I20260620 21:47:57.407579 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.395340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.1.155.189:38379)
0620 21:47:57.395744 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:57.395750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:57.395912 (+ 162us) server_negotiation.cc:407] Connection header received
0620 21:47:57.396127 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:57.396130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:57.396183 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:57.396241 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:57.397222 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.398007 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.399161 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.399479 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.399563 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.402513 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:57.402518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:57.402524 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:57.402557 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:57.404981 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.406202 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.406206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.406208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.406301 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.406614 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.406618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.406619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.406837 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:57.406941 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:57.407301 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 21:47:57.407420 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324}
W20260620 21:47:57.408192 15232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:47:57.408389 15232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:47:57.405289 15060 master_runner.cc:186] Error getting master registration for 127.1.155.190:45263: OK, instance_id { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" instance_seqno: 1781992076908859 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:47:57.408831 15060 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:47:57.410507 15053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } }
I20260620 21:47:57.410866 15053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:47:57.411015 15053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76da812ce0f9409d9ded0e6059c6baca, State: Initialized, Role: FOLLOWER
I20260620 21:47:57.411630 15053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } }
I20260620 21:47:57.413195 15053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:47:57.414222 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.394988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56360 (local address 127.1.155.190:45263)
0620 21:47:57.395129 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:57.395136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:57.400259 (+ 5123us) server_negotiation.cc:407] Connection header received
0620 21:47:57.401535 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:57.401538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:57.401593 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:57.401645 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:57.402889 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.403632 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.408345 (+ 4713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.408652 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.408738 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.410533 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:57.410538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:57.410541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:57.410569 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:57.412227 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.412750 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.412754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.412756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.412840 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.413146 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.413149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.413151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.413375 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:57.413487 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:57.413868 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 21:47:57.414012 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":36}
I20260620 21:47:57.416575 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:47:57.395536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40950 (local address 127.1.155.188:41171)
0620 21:47:57.399654 (+ 4118us) server_negotiation.cc:206] Beginning negotiation
0620 21:47:57.399658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:47:57.402076 (+ 2418us) server_negotiation.cc:407] Connection header received
0620 21:47:57.402190 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:47:57.402193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:47:57.402249 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:47:57.402331 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:47:57.403532 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.404570 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.410163 (+ 5593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:47:57.410523 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:47:57.410627 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:47:57.412749 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:47:57.412754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:47:57.412757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:47:57.412791 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:47:57.414519 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.415008 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.415014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.415017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.415102 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:47:57.415390 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:47:57.415393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:47:57.415395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:47:57.415618 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:47:57.415780 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 21:47:57.416310 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 21:47:57.416431 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4036}
I20260620 21:47:57.418700 15234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } } }
I20260620 21:47:57.418871 15234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:47:57.419927 15204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b: No bootstrap required, opened a new log
I20260620 21:47:57.424670 15204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } }
I20260620 21:47:57.424896 15204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:47:57.424979 15204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65fdad74142442d68800125606265d2b, State: Initialized, Role: FOLLOWER
I20260620 21:47:57.425498 15204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } }
I20260620 21:47:57.426072 15247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } } }
I20260620 21:47:57.426257 15247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:47:57.422582 14985 master_runner.cc:186] Error getting master registration for 127.1.155.188:41171: OK, instance_id { permanent_uuid: "65fdad74142442d68800125606265d2b" instance_seqno: 1781992077338269 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:47:57.426496 14985 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:47:57.426961 15204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b [sys.catalog]: configured and running, proceeding with master startup.
W20260620 21:47:57.428362 15245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:47:57.428453 15245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 21:47:57.433226 15258 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:47:57.433357 15258 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:47:57.436662 15134 master_runner.cc:263] No leader master found from master 65fdad74142442d68800125606265d2b
I20260620 21:47:57.436735 15134 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 21:47:57.584614 15157 tablet.cc:2369] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:47:57.609820 15081 tablet.cc:2369] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:47:57.655442 15006 tablet.cc:2369] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:47:57.679221 15234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:47:57.679458 15234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } }
I20260620 21:47:57.680364 15234 leader_election.cc:288] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7603ab9512a1497c894eca5b9c4423ec (127.1.155.189:38379), 65fdad74142442d68800125606265d2b (127.1.155.188:41171)
I20260620 21:47:57.680867 15102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76da812ce0f9409d9ded0e6059c6baca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7603ab9512a1497c894eca5b9c4423ec" is_pre_election: true
I20260620 21:47:57.680887 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76da812ce0f9409d9ded0e6059c6baca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65fdad74142442d68800125606265d2b" is_pre_election: true
I20260620 21:47:57.681211 15178 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76da812ce0f9409d9ded0e6059c6baca in term 0.
I20260620 21:47:57.681197 15102 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76da812ce0f9409d9ded0e6059c6baca in term 0.
I20260620 21:47:57.681668 15002 leader_election.cc:302] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7603ab9512a1497c894eca5b9c4423ec, 76da812ce0f9409d9ded0e6059c6baca; no voters:
I20260620 21:47:57.681983 15234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:47:57.682080 15234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:47:57.682138 15234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:47:57.683174 15234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } }
I20260620 21:47:57.683528 15234 leader_election.cc:288] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [CANDIDATE]: Term 1 election: Requested vote from peers 7603ab9512a1497c894eca5b9c4423ec (127.1.155.189:38379), 65fdad74142442d68800125606265d2b (127.1.155.188:41171)
I20260620 21:47:57.684016 15102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76da812ce0f9409d9ded0e6059c6baca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7603ab9512a1497c894eca5b9c4423ec"
I20260620 21:47:57.684016 15178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76da812ce0f9409d9ded0e6059c6baca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65fdad74142442d68800125606265d2b"
I20260620 21:47:57.684149 15178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:47:57.684149 15102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:47:57.685380 15102 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76da812ce0f9409d9ded0e6059c6baca in term 1.
I20260620 21:47:57.685381 15178 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76da812ce0f9409d9ded0e6059c6baca in term 1.
I20260620 21:47:57.685710 15004 leader_election.cc:302] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65fdad74142442d68800125606265d2b, 76da812ce0f9409d9ded0e6059c6baca; no voters:
I20260620 21:47:57.685925 15234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:47:57.686177 15234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [term 1 LEADER]: Becoming Leader. State: Replica: 76da812ce0f9409d9ded0e6059c6baca, State: Running, Role: LEADER
I20260620 21:47:57.686488 15234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } }
I20260620 21:47:57.688297 15260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76da812ce0f9409d9ded0e6059c6baca. Latest consensus state: current_term: 1 leader_uuid: "76da812ce0f9409d9ded0e6059c6baca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } } }
I20260620 21:47:57.688418 15260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [sys.catalog]: This master's current role is: LEADER
I20260620 21:47:57.688798 15262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:47:57.689608 15262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:47:57.694021 15178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b [term 1 FOLLOWER]: Refusing update from remote peer 76da812ce0f9409d9ded0e6059c6baca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:47:57.694064 15102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec [term 1 FOLLOWER]: Refusing update from remote peer 76da812ce0f9409d9ded0e6059c6baca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:47:57.694636 15234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:47:57.695041 15260 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [LEADER]: Connected to new peer: Peer: permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:47:57.698715 15247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76da812ce0f9409d9ded0e6059c6baca. Latest consensus state: current_term: 1 leader_uuid: "76da812ce0f9409d9ded0e6059c6baca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } } }
I20260620 21:47:57.698715 15220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76da812ce0f9409d9ded0e6059c6baca. Latest consensus state: current_term: 1 leader_uuid: "76da812ce0f9409d9ded0e6059c6baca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } } }
I20260620 21:47:57.698832 15247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:47:57.698832 15220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:47:57.700636 15234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76da812ce0f9409d9ded0e6059c6baca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } } }
I20260620 21:47:57.700752 15234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [sys.catalog]: This master's current role is: LEADER
I20260620 21:47:57.701471 15234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76da812ce0f9409d9ded0e6059c6baca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } } }
I20260620 21:47:57.701584 15234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca [sys.catalog]: This master's current role is: LEADER
I20260620 21:47:57.701814 15220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76da812ce0f9409d9ded0e6059c6baca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } } }
I20260620 21:47:57.701961 15220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:47:57.702237 15262 catalog_manager.cc:1347] Generated new cluster ID: d01a639ccd73442ea667a6dbe8392634
I20260620 21:47:57.702287 15262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:47:57.702994 15247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76da812ce0f9409d9ded0e6059c6baca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76da812ce0f9409d9ded0e6059c6baca" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45263 } } peers { permanent_uuid: "7603ab9512a1497c894eca5b9c4423ec" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 38379 } } peers { permanent_uuid: "65fdad74142442d68800125606265d2b" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 41171 } } }
I20260620 21:47:57.703099 15247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:47:57.717406 15262 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:47:57.718636 15262 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:47:57.726704 15262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76da812ce0f9409d9ded0e6059c6baca: Generated new TSK 0
I20260620 21:47:57.727615 15262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:47:58.409776 15232 catalog_manager.cc:1259] Loaded cluster ID: d01a639ccd73442ea667a6dbe8392634
I20260620 21:47:58.409940 15232 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec: loading cluster ID for follower catalog manager: success
I20260620 21:47:58.411563 15232 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec: acquiring CA information for follower catalog manager: success
I20260620 21:47:58.412678 15232 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7603ab9512a1497c894eca5b9c4423ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:47:58.434769 15258 catalog_manager.cc:1259] Loaded cluster ID: d01a639ccd73442ea667a6dbe8392634
I20260620 21:47:58.434907 15258 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b: loading cluster ID for follower catalog manager: success
I20260620 21:47:58.436601 15258 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b: acquiring CA information for follower catalog manager: success
I20260620 21:47:58.437721 15258 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65fdad74142442d68800125606265d2b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:47:58.798442 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 14985
I20260620 21:47:58.805017 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 15060
I20260620 21:47:58.810992 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 15134
2026-06-20T21:47:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3245 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:47:58 dist-test-slave-5bvn krb5kdc[15278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:47:58 dist-test-slave-5bvn krb5kdc[15278](info): set up 2 sockets
Jun 20 21:47:58 dist-test-slave-5bvn krb5kdc[15278](info): commencing operation
krb5kdc: starting...
W20260620 21:47:59.861982 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:47:59 dist-test-slave-5bvn krb5kdc[15278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:47:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:47:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:47:59.978927 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:35863
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:41909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:35863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:48:00.093302 15291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:00.093641 15291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:00.093722 15291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:00.097807 15291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:48:00.097925 15291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:00.097965 15291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:00.098002 15291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:48:00.098037 15291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:48:00.103274 15291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:41909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:35863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:35863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:00.104343 15291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:00.105360 15291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:00.113955 15297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:00.113960 15299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:00.113960 15296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:00.114348 15291 server_base.cc:1034] running on GCE node
I20260620 21:48:00.114952 15291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:00.116039 15291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:00.117242 15291 hybrid_clock.cc:648] HybridClock initialized: now 1781992080117205 us; error 47 us; skew 500 ppm
Jun 20 21:48:00 dist-test-slave-5bvn krb5kdc[15278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992080, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:00.120103 15291 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:48:00.121045 15291 webserver.cc:460] Webserver started at http://127.1.155.190:33627/ using document root <none> and password file <none>
I20260620 21:48:00.121646 15291 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:00.121739 15291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:00.122035 15291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:48:00.124106 15291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "15731814654f4782baa8caa5e5668d1a"
format_stamp: "Formatted at 2026-06-20 21:48:00 on dist-test-slave-5bvn"
server_key: "078f42e930374eadc7633da687686bc6"
server_key_iv: "7cd79a8062510b2ad091c26400e2c548"
server_key_version: "encryptionkey@0"
I20260620 21:48:00.124742 15291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "15731814654f4782baa8caa5e5668d1a"
format_stamp: "Formatted at 2026-06-20 21:48:00 on dist-test-slave-5bvn"
server_key: "078f42e930374eadc7633da687686bc6"
server_key_iv: "7cd79a8062510b2ad091c26400e2c548"
server_key_version: "encryptionkey@0"
I20260620 21:48:00.128696 15291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 21:48:00.131366 15306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:00.132717 15291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:48:00.132897 15291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "15731814654f4782baa8caa5e5668d1a"
format_stamp: "Formatted at 2026-06-20 21:48:00 on dist-test-slave-5bvn"
server_key: "078f42e930374eadc7633da687686bc6"
server_key_iv: "7cd79a8062510b2ad091c26400e2c548"
server_key_version: "encryptionkey@0"
I20260620 21:48:00.133025 15291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:00.147336 15291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:00.150852 15291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:00.151068 15291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:00.157898 15291 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:35863
I20260620 21:48:00.157939 15358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:35863 every 8 connection(s)
I20260620 21:48:00.158875 15291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:48:00.161562 15359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:48:00.165302 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 15291
I20260620 21:48:00.165448 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:48:00.165711 1646 external_mini_cluster.cc:1428] Setting key 2da568c31a1d6487ed49178cad4241ec
I20260620 21:48:00.167732 15359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a: Bootstrap starting.
I20260620 21:48:00.170475 15359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a: Neither blocks nor log segments found. Creating new log.
I20260620 21:48:00.171353 15359 log.cc:824] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:00.173520 15359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a: No bootstrap required, opened a new log
Jun 20 21:48:00 dist-test-slave-5bvn krb5kdc[15278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:48:00.178519 15359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15731814654f4782baa8caa5e5668d1a" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35863 } }
I20260620 21:48:00.178786 15359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:48:00.178848 15359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15731814654f4782baa8caa5e5668d1a, State: Initialized, Role: FOLLOWER
I20260620 21:48:00.179385 15359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15731814654f4782baa8caa5e5668d1a" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35863 } }
I20260620 21:48:00.179565 15359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:48:00.179638 15359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:48:00.179734 15359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:48:00.180948 15359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15731814654f4782baa8caa5e5668d1a" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35863 } }
I20260620 21:48:00.181322 15359 leader_election.cc:302] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15731814654f4782baa8caa5e5668d1a; no voters:
I20260620 21:48:00.181648 15359 leader_election.cc:288] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:48:00.181778 15364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:48:00.181975 15362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:00.167092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.1.155.190:35863)
0620 21:48:00.167528 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:00.167547 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:00.167589 (+ 42us) server_negotiation.cc:407] Connection header received
0620 21:48:00.168389 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:00.168411 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:00.168838 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:00.169214 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:00.169961 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:00.171502 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:00.172365 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:00.173030 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:00.173188 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:00.176101 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:00.176108 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:00.176126 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:00.176159 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:00.178315 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:00.179087 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:00.179093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:00.179099 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:00.179234 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:00.179521 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:00.179524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:00.179526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:00.180102 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:00.180291 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:00.180540 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 21:48:00.180878 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":132,"threads_started":1}
I20260620 21:48:00.182040 15364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [term 1 LEADER]: Becoming Leader. State: Replica: 15731814654f4782baa8caa5e5668d1a, State: Running, Role: LEADER
I20260620 21:48:00.182838 15364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15731814654f4782baa8caa5e5668d1a" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35863 } }
I20260620 21:48:00.182969 15359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:48:00.183969 15364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15731814654f4782baa8caa5e5668d1a. Latest consensus state: current_term: 1 leader_uuid: "15731814654f4782baa8caa5e5668d1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15731814654f4782baa8caa5e5668d1a" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35863 } } }
I20260620 21:48:00.184096 15364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [sys.catalog]: This master's current role is: LEADER
I20260620 21:48:00.184417 15366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15731814654f4782baa8caa5e5668d1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15731814654f4782baa8caa5e5668d1a" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 35863 } } }
I20260620 21:48:00.184510 15366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a [sys.catalog]: This master's current role is: LEADER
I20260620 21:48:00.184818 15373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:48:00.185463 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:48:00.188012 15373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:48:00.193979 15373 catalog_manager.cc:1347] Generated new cluster ID: d44347ee6cbb41478571d9ec89abb358
I20260620 21:48:00.194056 15373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:48:00.235530 15373 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:48:00.236831 15373 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:48:00.245266 15373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15731814654f4782baa8caa5e5668d1a: Generated new TSK 0
I20260620 21:48:00.246137 15373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:48:00.292522 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 15291
2026-06-20T21:48:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1479 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:48:00 dist-test-slave-5bvn krb5kdc[15386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:48:00 dist-test-slave-5bvn krb5kdc[15386](info): set up 2 sockets
Jun 20 21:48:00 dist-test-slave-5bvn krb5kdc[15386](info): commencing operation
krb5kdc: starting...
W20260620 21:48:01.334177 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:48:01 dist-test-slave-5bvn krb5kdc[15386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:48:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:48:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:48:01.452929 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:39637
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:39637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:48:01.563807 15399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:01.564098 15399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:01.564169 15399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:01.568192 15399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:48:01.568271 15399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:01.568307 15399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:01.568344 15399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:48:01.568377 15399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:48:01.573514 15399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:39637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:39637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:01.574579 15399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:01.575577 15399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:01.584399 15407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:01.584400 15405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:01.584431 15404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:01.584444 15399 server_base.cc:1034] running on GCE node
I20260620 21:48:01.585234 15399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:01.586440 15399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:01.587637 15399 hybrid_clock.cc:648] HybridClock initialized: now 1781992081587615 us; error 35 us; skew 500 ppm
Jun 20 21:48:01 dist-test-slave-5bvn krb5kdc[15386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992081, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:01.595713 15399 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:48:01.596635 15399 webserver.cc:460] Webserver started at http://127.1.155.190:35275/ using document root <none> and password file <none>
I20260620 21:48:01.597155 15399 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:01.597211 15399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:01.597393 15399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:48:01.599476 15399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "de3d7860a515454f80ea87f3cb31b634"
format_stamp: "Formatted at 2026-06-20 21:48:01 on dist-test-slave-5bvn"
server_key: "8da1e54d49ce66ce64dccf4ea60c8806"
server_key_iv: "53a67a1b67ff3be7f37c7892c6f0592c"
server_key_version: "encryptionkey@0"
I20260620 21:48:01.599983 15399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "de3d7860a515454f80ea87f3cb31b634"
format_stamp: "Formatted at 2026-06-20 21:48:01 on dist-test-slave-5bvn"
server_key: "8da1e54d49ce66ce64dccf4ea60c8806"
server_key_iv: "53a67a1b67ff3be7f37c7892c6f0592c"
server_key_version: "encryptionkey@0"
I20260620 21:48:01.603693 15399 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:48:01.606176 15414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:01.607488 15399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:48:01.607623 15399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "de3d7860a515454f80ea87f3cb31b634"
format_stamp: "Formatted at 2026-06-20 21:48:01 on dist-test-slave-5bvn"
server_key: "8da1e54d49ce66ce64dccf4ea60c8806"
server_key_iv: "53a67a1b67ff3be7f37c7892c6f0592c"
server_key_version: "encryptionkey@0"
I20260620 21:48:01.607733 15399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:01.635063 15399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:01.638177 15399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:01.638388 15399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:01.645323 15399 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:39637
I20260620 21:48:01.645606 15466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:39637 every 8 connection(s)
I20260620 21:48:01.646340 15399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:48:01.649040 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:48:01.649422 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 15399
I20260620 21:48:01.649566 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:48:01.649919 1646 external_mini_cluster.cc:1428] Setting key a78bcf6763e44ce44ef6e5648c26a22c
I20260620 21:48:01.656229 15467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: Bootstrap starting.
I20260620 21:48:01.659138 15467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: Neither blocks nor log segments found. Creating new log.
Jun 20 21:48:01 dist-test-slave-5bvn krb5kdc[15386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:48:01.659952 15467 log.cc:824] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:01.662045 15467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: No bootstrap required, opened a new log
I20260620 21:48:01.665467 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:01.651451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.1.155.190:39637)
0620 21:48:01.651897 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:01.651911 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:01.651945 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:48:01.652568 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:01.652584 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:01.652903 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:01.653167 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:01.654087 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:01.655060 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:01.656270 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:01.656876 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:01.657013 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:01.660703 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:01.660708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:01.660727 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:01.660760 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:01.662645 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:01.663120 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:01.663131 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:01.663136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:01.663267 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:01.663564 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:01.663569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:01.663571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:01.663965 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:01.664166 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:01.664448 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 21:48:01.664732 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":128,"threads_started":1}
I20260620 21:48:01.665586 15467 raft_consensus.cc:348] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } }
I20260620 21:48:01.665895 15467 raft_consensus.cc:374] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:48:01.665988 15467 raft_consensus.cc:729] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3d7860a515454f80ea87f3cb31b634, State: Initialized, Role: FOLLOWER
I20260620 21:48:01.666469 15467 consensus_queue.cc:260] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } }
I20260620 21:48:01.666623 15467 raft_consensus.cc:388] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:48:01.666711 15467 raft_consensus.cc:482] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:48:01.666808 15467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:48:01.667770 15467 raft_consensus.cc:504] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } }
I20260620 21:48:01.668111 15467 leader_election.cc:302] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de3d7860a515454f80ea87f3cb31b634; no voters:
I20260620 21:48:01.668447 15467 leader_election.cc:288] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:48:01.668555 15472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:48:01.668787 15472 raft_consensus.cc:686] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 1 LEADER]: Becoming Leader. State: Replica: de3d7860a515454f80ea87f3cb31b634, State: Running, Role: LEADER
I20260620 21:48:01.669109 15472 consensus_queue.cc:237] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } }
I20260620 21:48:01.669632 15467 sys_catalog.cc:564] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:48:01.670806 15474 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de3d7860a515454f80ea87f3cb31b634. Latest consensus state: current_term: 1 leader_uuid: "de3d7860a515454f80ea87f3cb31b634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } } }
I20260620 21:48:01.670818 15473 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de3d7860a515454f80ea87f3cb31b634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } } }
I20260620 21:48:01.670935 15474 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [sys.catalog]: This master's current role is: LEADER
I20260620 21:48:01.670990 15473 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [sys.catalog]: This master's current role is: LEADER
I20260620 21:48:01.671378 15481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:48:01.674139 15481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:48:01.679857 15481 catalog_manager.cc:1347] Generated new cluster ID: 6abebb26bcb2422e88efde2e9161669a
I20260620 21:48:01.679929 15481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:48:01.697454 15481 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:48:01.698515 15481 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:48:01.703454 15481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: Generated new TSK 0
I20260620 21:48:01.704183 15481 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:48:01.743085 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39637
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:48:01.857640 15493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:01.857991 15493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:01.858153 15493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:01.862541 15493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:01.862612 15493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:01.862674 15493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:48:01.868079 15493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:01.869238 15493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:01.870339 15493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:01.880173 15501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:01.880214 15493 server_base.cc:1034] running on GCE node
W20260620 21:48:01.880241 15498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:01.880241 15499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:01.880822 15493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:01.881441 15493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:01.882601 15493 hybrid_clock.cc:648] HybridClock initialized: now 1781992081882586 us; error 26 us; skew 500 ppm
Jun 20 21:48:01 dist-test-slave-5bvn krb5kdc[15386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992081, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:01.885306 15493 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:48:01.886252 15493 webserver.cc:460] Webserver started at http://127.1.155.129:33461/ using document root <none> and password file <none>
I20260620 21:48:01.886816 15493 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:01.886931 15493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:01.887173 15493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:48:01.889184 15493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "89d4f97e5c28415d9633552e764362a5"
format_stamp: "Formatted at 2026-06-20 21:48:01 on dist-test-slave-5bvn"
server_key: "2c86143ff3fac0b5734504aceb439007"
server_key_iv: "722892b9e377e8877d0e28d173a193cc"
server_key_version: "encryptionkey@0"
I20260620 21:48:01.889823 15493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "89d4f97e5c28415d9633552e764362a5"
format_stamp: "Formatted at 2026-06-20 21:48:01 on dist-test-slave-5bvn"
server_key: "2c86143ff3fac0b5734504aceb439007"
server_key_iv: "722892b9e377e8877d0e28d173a193cc"
server_key_version: "encryptionkey@0"
I20260620 21:48:01.893532 15493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:48:01.895910 15508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:01.897099 15493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:48:01.897235 15493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "89d4f97e5c28415d9633552e764362a5"
format_stamp: "Formatted at 2026-06-20 21:48:01 on dist-test-slave-5bvn"
server_key: "2c86143ff3fac0b5734504aceb439007"
server_key_iv: "722892b9e377e8877d0e28d173a193cc"
server_key_version: "encryptionkey@0"
I20260620 21:48:01.897346 15493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:01.908155 15493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:01.911530 15493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:01.911759 15493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:01.912463 15493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:48:01.913337 15493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:48:01.913416 15493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:01.913494 15493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:48:01.913542 15493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:01.923576 15493 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:34437
I20260620 21:48:01.923593 15621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:34437 every 8 connection(s)
I20260620 21:48:01.924525 15493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:48:01.929785 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 15493
I20260620 21:48:01.929927 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:48:01.930222 1646 external_mini_cluster.cc:1428] Setting key 06ac3e15d9d0ea9f596f2e86c169ba2d
Jun 20 21:48:01 dist-test-slave-5bvn krb5kdc[15386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992081, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
I20260620 21:48:01.939239 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:01.926383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:32791 (local address 127.1.155.190:39637)
0620 21:48:01.926537 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:01.926542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:01.927492 (+ 950us) server_negotiation.cc:407] Connection header received
0620 21:48:01.928779 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:01.928782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:01.928861 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:01.928941 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:01.930331 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:01.931052 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:01.932387 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:01.932637 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:01.932754 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:01.935475 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:01.935479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:01.935481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:01.935515 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:01.937015 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:01.937576 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:01.937581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:01.937583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:01.937670 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:01.938092 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:01.938097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:01.938099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:01.938292 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:01.938404 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:01.938926 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 21:48:01.939052 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:48:01.940053 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:01.926837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:32791)
0620 21:48:01.927319 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:01.927362 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:01.928561 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:01.929114 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:01.929122 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:01.929514 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:01.930166 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:01.930183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:01.931185 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:01.931188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:01.932259 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:01.932269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:01.932794 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:01.932798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:01.932972 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:01.933596 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:01.933612 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:01.935303 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:01.937157 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:01.937163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:01.937179 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:01.937470 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:01.937767 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:01.937771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:01.937773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:01.937994 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:01.938403 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:01.938409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:01.938673 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 21:48:01.938861 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 21:48:01.939197 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":116,"threads_started":1}
I20260620 21:48:01.941419 15622 heartbeater.cc:347] Connected to a master server at 127.1.155.190:39637
I20260620 21:48:01.941699 15622 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
I20260620 21:48:01.942395 15622 heartbeater.cc:511] Master 127.1.155.190:39637 requested a full tablet report, sending...
I20260620 21:48:01.944092 15431 ts_manager.cc:194] Registered new tserver with Master: 89d4f97e5c28415d9633552e764362a5 (127.1.155.129:34437)
I20260620 21:48:01.945245 15431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:32791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:48:01.957270 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39637
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:48:02.068197 15627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:02.068475 15627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:02.068518 15627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:02.072870 15627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:02.072934 15627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:02.072994 15627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:48:02.078290 15627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:02.079355 15627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:02.080399 15627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:02.089910 15635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:02.089932 15632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:02.090087 15627 server_base.cc:1034] running on GCE node
W20260620 21:48:02.090420 15633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:02.090793 15627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:02.091404 15627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:02.092620 15627 hybrid_clock.cc:648] HybridClock initialized: now 1781992082092603 us; error 30 us; skew 500 ppm
Jun 20 21:48:02 dist-test-slave-5bvn krb5kdc[15386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992082, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:02.095504 15627 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:48:02.096371 15627 webserver.cc:460] Webserver started at http://127.1.155.130:34547/ using document root <none> and password file <none>
I20260620 21:48:02.096956 15627 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:02.097019 15627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:02.097257 15627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:48:02.099347 15627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "44286227758740a0944e5091566231b8"
format_stamp: "Formatted at 2026-06-20 21:48:02 on dist-test-slave-5bvn"
server_key: "fab0e3d7bff612e7b4793dda3633cfda"
server_key_iv: "e0f59244ff22b66788ec5293c5a6dd25"
server_key_version: "encryptionkey@0"
I20260620 21:48:02.099921 15627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "44286227758740a0944e5091566231b8"
format_stamp: "Formatted at 2026-06-20 21:48:02 on dist-test-slave-5bvn"
server_key: "fab0e3d7bff612e7b4793dda3633cfda"
server_key_iv: "e0f59244ff22b66788ec5293c5a6dd25"
server_key_version: "encryptionkey@0"
I20260620 21:48:02.104135 15627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:48:02.106812 15642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:02.108381 15627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:48:02.108528 15627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "44286227758740a0944e5091566231b8"
format_stamp: "Formatted at 2026-06-20 21:48:02 on dist-test-slave-5bvn"
server_key: "fab0e3d7bff612e7b4793dda3633cfda"
server_key_iv: "e0f59244ff22b66788ec5293c5a6dd25"
server_key_version: "encryptionkey@0"
I20260620 21:48:02.108650 15627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:02.140980 15627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:02.144786 15627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:02.145013 15627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:02.145613 15627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:48:02.146497 15627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:48:02.146576 15627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:02.146641 15627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:48:02.146692 15627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:02.155675 15627 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:34019
I20260620 21:48:02.155748 15755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:34019 every 8 connection(s)
I20260620 21:48:02.156606 15627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:48:02.164559 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 15627
I20260620 21:48:02.164716 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:48:02.165035 1646 external_mini_cluster.cc:1428] Setting key d09ac9fd95dc38cd9e5317f01c19e5f0
Jun 20 21:48:02 dist-test-slave-5bvn krb5kdc[15386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992082, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:48:02.170969 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.158461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:57665 (local address 127.1.155.190:39637)
0620 21:48:02.158614 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:02.158618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:02.159558 (+ 940us) server_negotiation.cc:407] Connection header received
0620 21:48:02.160811 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:02.160814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:02.160889 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:02.160967 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:02.162179 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.162706 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.163799 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.164064 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.164159 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.167072 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:02.167075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:02.167077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:02.167100 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:02.168566 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:02.169095 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:02.169098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:02.169100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:02.169185 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:02.169560 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:02.169566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:02.169572 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:02.169814 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:02.169947 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:02.170545 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 21:48:02.170757 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:48:02.172199 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.158902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:57665)
0620 21:48:02.159387 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:02.159442 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:02.160607 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:02.161142 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:02.161149 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:02.161497 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:02.162042 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:02.162057 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.162811 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.162814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:02.163675 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:02.163683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.164232 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.164235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:02.164379 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.165156 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:02.165173 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:02.166900 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:02.168700 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:02.168706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:02.168722 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:02.168987 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:02.169285 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:02.169288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:02.169290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:02.169441 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:02.169955 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:02.169964 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:02.170251 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 21:48:02.170477 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 21:48:02.170934 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":122,"threads_started":1}
I20260620 21:48:02.173471 15756 heartbeater.cc:347] Connected to a master server at 127.1.155.190:39637
I20260620 21:48:02.173719 15756 heartbeater.cc:464] Registering TS with master...
I20260620 21:48:02.174268 15756 heartbeater.cc:511] Master 127.1.155.190:39637 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
I20260620 21:48:02.175410 15431 ts_manager.cc:194] Registered new tserver with Master: 44286227758740a0944e5091566231b8 (127.1.155.130:34019)
I20260620 21:48:02.176084 15431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:57665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:48:02.193682 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39637
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:48:02.304018 15761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:02.304309 15761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:02.304409 15761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:02.308549 15761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:02.308647 15761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:02.308735 15761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:48:02.314452 15761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:02.315572 15761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:02.316572 15761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:02.326119 15769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:02.326083 15767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:02.326110 15766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:02.326395 15761 server_base.cc:1034] running on GCE node
I20260620 21:48:02.326882 15761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:02.327507 15761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:02.328707 15761 hybrid_clock.cc:648] HybridClock initialized: now 1781992082328693 us; error 28 us; skew 500 ppm
Jun 20 21:48:02 dist-test-slave-5bvn krb5kdc[15386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992082, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:02.336177 15761 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:48:02.337080 15761 webserver.cc:460] Webserver started at http://127.1.155.131:37519/ using document root <none> and password file <none>
I20260620 21:48:02.337682 15761 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:02.337779 15761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:02.338027 15761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:48:02.340122 15761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "e4347d5750a24d21a19dbd71663b08bc"
format_stamp: "Formatted at 2026-06-20 21:48:02 on dist-test-slave-5bvn"
server_key: "7874a4db27361a231224b9ab50efe95c"
server_key_iv: "2343f9ca95956857af8ee604dd95336e"
server_key_version: "encryptionkey@0"
I20260620 21:48:02.340670 15761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "e4347d5750a24d21a19dbd71663b08bc"
format_stamp: "Formatted at 2026-06-20 21:48:02 on dist-test-slave-5bvn"
server_key: "7874a4db27361a231224b9ab50efe95c"
server_key_iv: "2343f9ca95956857af8ee604dd95336e"
server_key_version: "encryptionkey@0"
I20260620 21:48:02.344424 15761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 21:48:02.346899 15776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:02.348186 15761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:48:02.348340 15761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "e4347d5750a24d21a19dbd71663b08bc"
format_stamp: "Formatted at 2026-06-20 21:48:02 on dist-test-slave-5bvn"
server_key: "7874a4db27361a231224b9ab50efe95c"
server_key_iv: "2343f9ca95956857af8ee604dd95336e"
server_key_version: "encryptionkey@0"
I20260620 21:48:02.348465 15761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:02.367897 15761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:02.371191 15761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:02.371397 15761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:02.372035 15761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:48:02.372867 15761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:48:02.372941 15761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:02.373006 15761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:48:02.373062 15761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:02.382135 15761 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:45423
I20260620 21:48:02.382207 15889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:45423 every 8 connection(s)
I20260620 21:48:02.383059 15761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:48:02.390419 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 15761
I20260620 21:48:02.390556 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:48:02.390822 1646 external_mini_cluster.cc:1428] Setting key 525e8ef10d1c3009380e93817ac5c376
Jun 20 21:48:02 dist-test-slave-5bvn krb5kdc[15386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992082, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:48:02.396862 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.384845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:49009 (local address 127.1.155.190:39637)
0620 21:48:02.385010 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:02.385015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:02.385822 (+ 807us) server_negotiation.cc:407] Connection header received
0620 21:48:02.387187 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:02.387190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:02.387237 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:02.387315 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:02.388557 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.389095 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.390261 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.390471 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.390622 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.393378 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:02.393382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:02.393384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:02.393408 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:02.394802 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:02.395445 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:02.395448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:02.395449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:02.395533 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:02.395871 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:02.395873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:02.395875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:02.396090 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:02.396171 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:02.396642 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 21:48:02.396743 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:48:02.397719 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.385231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:49009)
0620 21:48:02.385667 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:02.385709 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:02.386974 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:02.387477 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:02.387483 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:02.387825 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:02.388374 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:02.388389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.389217 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.389221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:02.390113 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:02.390123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.390677 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.390681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:02.390920 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.391635 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:02.391651 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:02.393229 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:02.395019 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:02.395025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:02.395037 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:02.395303 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:02.395636 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:02.395641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:02.395643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:02.395778 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:02.396185 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:02.396191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:02.396432 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 21:48:02.396591 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 21:48:02.396840 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260620 21:48:02.398933 15890 heartbeater.cc:347] Connected to a master server at 127.1.155.190:39637
I20260620 21:48:02.399145 15890 heartbeater.cc:464] Registering TS with master...
I20260620 21:48:02.399610 15890 heartbeater.cc:511] Master 127.1.155.190:39637 requested a full tablet report, sending...
I20260620 21:48:02.400863 15431 ts_manager.cc:194] Registered new tserver with Master: e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423)
I20260620 21:48:02.401499 15431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:49009
I20260620 21:48:02.404886 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:48:02.415128 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.407149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.1.155.190:39637)
0620 21:48:02.407293 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:02.407297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:02.407488 (+ 191us) server_negotiation.cc:407] Connection header received
0620 21:48:02.407589 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:02.407591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:02.407635 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:02.407709 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:02.408599 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.409120 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.410105 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.410314 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.410395 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.412039 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:02.412045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:02.412051 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:02.412080 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:02.413497 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:02.413988 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:02.413991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:02.413993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:02.414073 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:02.414362 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:02.414365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:02.414367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:02.414589 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:02.414695 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:02.414878 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 21:48:02.415002 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:48:02.417914 15431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:48:02.420890 15431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:48:02.436035 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.429597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:45423 (local address 127.0.0.1:45736)
0620 21:48:02.430011 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:02.430067 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:02.430210 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:02.431233 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:02.431239 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:02.431261 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:02.431419 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:02.431426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.433139 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.433143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:02.434544 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:02.434555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.435593 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.435597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:02.435689 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.435710 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:48:02.435769 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 21:48:02.435865 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":48,"thread_start_us":99,"threads_started":1}
I20260620 21:48:02.436102 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.430772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.1.155.129:34437)
0620 21:48:02.431115 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:02.431130 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:02.431153 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:48:02.431229 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:02.431233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:02.431403 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:02.431575 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:02.431832 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.432965 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.434626 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.435462 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.435570 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.435709 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:02.435867 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 21:48:02.435968 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":129,"threads_started":1}
I20260620 21:48:02.436168 15905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.430407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.1.155.130:34019)
0620 21:48:02.431214 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:02.431227 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:02.431253 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:48:02.431331 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:02.431339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:02.431562 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:02.431978 (+ 416us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:02.432101 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.432955 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.434734 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.435279 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.435410 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.435562 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:02.435950 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 21:48:02.436026 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":115,"threads_started":1}
I20260620 21:48:02.436473 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.430016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:34019 (local address 127.0.0.1:53718)
0620 21:48:02.430618 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:02.430633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:02.430726 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:02.431712 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:02.431714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:02.431729 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:02.431830 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:02.431844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.433078 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.433085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:02.434605 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:02.434615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.435410 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.435414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:02.435537 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.435575 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 21:48:02.435683 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 21:48:02.435811 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":187,"thread_start_us":78,"threads_started":1}
I20260620 21:48:02.436486 15903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.430349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:34437 (local address 127.0.0.1:40582)
0620 21:48:02.430847 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:02.430861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:02.430957 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:02.431562 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:02.431566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:02.431583 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:02.431708 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:02.431714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.433121 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.433123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:02.434504 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:02.434513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.435640 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.435646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:02.435732 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.435751 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 21:48:02.435820 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 21:48:02.435875 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":152,"thread_start_us":101,"threads_started":1}
I20260620 21:48:02.437069 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.429762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.1.155.131:45423)
0620 21:48:02.430711 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:02.430723 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:02.430748 (+ 25us) server_negotiation.cc:407] Connection header received
0620 21:48:02.430828 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:02.430833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:02.431053 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:02.431576 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:02.431734 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.432964 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.434674 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.435452 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.436629 (+ 1177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.436786 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:02.436832 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 21:48:02.436937 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":192,"threads_started":1}
I20260620 21:48:02.439842 15556 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed6f9d389e546cea6f5f791c1e4a12c (DEFAULT_TABLE table=test-table [id=9cdc308fded94770bbb6bdb0733725a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:48:02.441146 15556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed6f9d389e546cea6f5f791c1e4a12c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:48:02.441040 15824 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed6f9d389e546cea6f5f791c1e4a12c (DEFAULT_TABLE table=test-table [id=9cdc308fded94770bbb6bdb0733725a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:48:02.442525 15824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed6f9d389e546cea6f5f791c1e4a12c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:48:02.440017 15690 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed6f9d389e546cea6f5f791c1e4a12c (DEFAULT_TABLE table=test-table [id=9cdc308fded94770bbb6bdb0733725a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:48:02.443809 15690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed6f9d389e546cea6f5f791c1e4a12c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:48:02.448897 15907 tablet_bootstrap.cc:492] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: Bootstrap starting.
I20260620 21:48:02.450894 15908 tablet_bootstrap.cc:492] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc: Bootstrap starting.
I20260620 21:48:02.451349 15907 tablet_bootstrap.cc:654] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: Neither blocks nor log segments found. Creating new log.
I20260620 21:48:02.452342 15909 tablet_bootstrap.cc:492] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8: Bootstrap starting.
I20260620 21:48:02.452479 15907 log.cc:824] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:02.453661 15908 tablet_bootstrap.cc:654] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc: Neither blocks nor log segments found. Creating new log.
I20260620 21:48:02.454593 15908 log.cc:824] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:02.455410 15909 tablet_bootstrap.cc:654] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8: Neither blocks nor log segments found. Creating new log.
I20260620 21:48:02.456341 15909 log.cc:824] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:02.456804 15908 tablet_bootstrap.cc:492] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc: No bootstrap required, opened a new log
I20260620 21:48:02.457012 15908 ts_tablet_manager.cc:1397] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:48:02.458494 15907 tablet_bootstrap.cc:492] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: No bootstrap required, opened a new log
I20260620 21:48:02.458618 15909 tablet_bootstrap.cc:492] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8: No bootstrap required, opened a new log
I20260620 21:48:02.458701 15907 ts_tablet_manager.cc:1397] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260620 21:48:02.458809 15909 ts_tablet_manager.cc:1397] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260620 21:48:02.461108 15908 raft_consensus.cc:348] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } } peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } }
I20260620 21:48:02.461392 15908 raft_consensus.cc:374] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:48:02.461448 15908 raft_consensus.cc:729] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4347d5750a24d21a19dbd71663b08bc, State: Initialized, Role: FOLLOWER
I20260620 21:48:02.461936 15908 consensus_queue.cc:260] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } } peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } }
I20260620 21:48:02.462955 15908 ts_tablet_manager.cc:1428] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:48:02.462970 15907 raft_consensus.cc:348] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } } peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } }
I20260620 21:48:02.463198 15907 raft_consensus.cc:374] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:48:02.463003 15909 raft_consensus.cc:348] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } } peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } }
I20260620 21:48:02.463245 15907 raft_consensus.cc:729] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d4f97e5c28415d9633552e764362a5, State: Initialized, Role: FOLLOWER
I20260620 21:48:02.463268 15909 raft_consensus.cc:374] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:48:02.463318 15909 raft_consensus.cc:729] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44286227758740a0944e5091566231b8, State: Initialized, Role: FOLLOWER
I20260620 21:48:02.463325 15890 heartbeater.cc:503] Master 127.1.155.190:39637 was elected leader, sending a full tablet report...
I20260620 21:48:02.463658 15907 consensus_queue.cc:260] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } } peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } }
I20260620 21:48:02.463781 15909 consensus_queue.cc:260] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } } peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } }
I20260620 21:48:02.464573 15907 ts_tablet_manager.cc:1428] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:48:02.464737 15909 ts_tablet_manager.cc:1428] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 21:48:02.464883 15622 heartbeater.cc:503] Master 127.1.155.190:39637 was elected leader, sending a full tablet report...
I20260620 21:48:02.465058 15756 heartbeater.cc:503] Master 127.1.155.190:39637 was elected leader, sending a full tablet report...
W20260620 21:48:02.634099 15891 tablet.cc:2369] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:48:02.657744 15757 tablet.cc:2369] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:48:02.676219 15623 tablet.cc:2369] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:48:02.717499 15914 raft_consensus.cc:482] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:48:02.717722 15914 raft_consensus.cc:504] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } } peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } }
I20260620 21:48:02.718916 15914 leader_election.cc:288] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423), 44286227758740a0944e5091566231b8 (127.1.155.130:34019)
I20260620 21:48:02.723510 15916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.719388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:34019 (local address 127.1.155.129:37641)
0620 21:48:02.719828 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:02.719850 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:02.719956 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:02.720310 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:02.720313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:02.720331 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:02.720469 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:02.720478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.721423 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.721425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:02.722360 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:02.722370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.723036 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.723040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:02.723136 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.723163 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:48:02.723253 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 21:48:02.723330 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":21,"thread_start_us":134,"threads_started":1}
I20260620 21:48:02.723575 15905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.719605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:37641 (local address 127.1.155.130:34019)
0620 21:48:02.719747 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:02.719753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:02.720038 (+ 285us) server_negotiation.cc:407] Connection header received
0620 21:48:02.720111 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:02.720117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:02.720182 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:02.720270 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:02.720611 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.721273 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.722491 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.722923 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.723016 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.723058 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:02.723336 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 21:48:02.723421 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:48:02.723873 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.719618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:60183 (local address 127.1.155.131:45423)
0620 21:48:02.719786 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:02.719791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:02.720224 (+ 433us) server_negotiation.cc:407] Connection header received
0620 21:48:02.720343 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:02.720346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:02.720390 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:02.720495 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:02.720834 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.721461 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.722605 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.723212 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.723319 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.723361 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:02.723666 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 21:48:02.723752 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 21:48:02.724216 15917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.719388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:45423 (local address 127.1.155.129:60183)
0620 21:48:02.720083 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:02.720096 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:02.720241 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:02.720528 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:02.720531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:02.720548 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:02.720702 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:02.720710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.721575 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.721579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:02.722481 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:02.722491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.723348 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.723352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:02.723481 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.723508 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:48:02.724032 (+ 524us) client_negotiation.cc:240] Negotiation successful
0620 21:48:02.724095 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":52,"thread_start_us":99,"threads_started":1}
I20260620 21:48:02.724301 15710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed6f9d389e546cea6f5f791c1e4a12c" candidate_uuid: "89d4f97e5c28415d9633552e764362a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44286227758740a0944e5091566231b8" is_pre_election: true
I20260620 21:48:02.724622 15710 raft_consensus.cc:2454] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89d4f97e5c28415d9633552e764362a5 in term 0.
I20260620 21:48:02.724922 15844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed6f9d389e546cea6f5f791c1e4a12c" candidate_uuid: "89d4f97e5c28415d9633552e764362a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4347d5750a24d21a19dbd71663b08bc" is_pre_election: true
I20260620 21:48:02.725150 15511 leader_election.cc:302] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44286227758740a0944e5091566231b8, 89d4f97e5c28415d9633552e764362a5; no voters:
I20260620 21:48:02.725267 15844 raft_consensus.cc:2454] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89d4f97e5c28415d9633552e764362a5 in term 0.
I20260620 21:48:02.725482 15914 raft_consensus.cc:2781] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:48:02.725598 15914 raft_consensus.cc:482] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:48:02.725687 15914 raft_consensus.cc:3037] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:48:02.726819 15914 raft_consensus.cc:504] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } } peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } }
I20260620 21:48:02.727211 15914 leader_election.cc:288] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 1 election: Requested vote from peers e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423), 44286227758740a0944e5091566231b8 (127.1.155.130:34019)
I20260620 21:48:02.727568 15844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed6f9d389e546cea6f5f791c1e4a12c" candidate_uuid: "89d4f97e5c28415d9633552e764362a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4347d5750a24d21a19dbd71663b08bc"
I20260620 21:48:02.727685 15844 raft_consensus.cc:3037] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:48:02.727696 15710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed6f9d389e546cea6f5f791c1e4a12c" candidate_uuid: "89d4f97e5c28415d9633552e764362a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44286227758740a0944e5091566231b8"
I20260620 21:48:02.727856 15710 raft_consensus.cc:3037] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:48:02.728854 15844 raft_consensus.cc:2454] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89d4f97e5c28415d9633552e764362a5 in term 1.
I20260620 21:48:02.729038 15710 raft_consensus.cc:2454] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89d4f97e5c28415d9633552e764362a5 in term 1.
I20260620 21:48:02.729215 15509 leader_election.cc:302] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d4f97e5c28415d9633552e764362a5, e4347d5750a24d21a19dbd71663b08bc; no voters:
I20260620 21:48:02.729413 15914 raft_consensus.cc:2781] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:48:02.729655 15914 raft_consensus.cc:686] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [term 1 LEADER]: Becoming Leader. State: Replica: 89d4f97e5c28415d9633552e764362a5, State: Running, Role: LEADER
I20260620 21:48:02.730024 15914 consensus_queue.cc:237] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } } peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } }
I20260620 21:48:02.733163 15430 catalog_manager.cc:5495] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89d4f97e5c28415d9633552e764362a5 (127.1.155.129). New cstate: current_term: 1 leader_uuid: "89d4f97e5c28415d9633552e764362a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } health_report { overall_health: UNKNOWN } } }
I20260620 21:48:02.800911 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.797066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40598 (local address 127.1.155.129:34437)
0620 21:48:02.797208 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:02.797212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:02.797224 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:48:02.797298 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:02.797300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:02.797352 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:02.797442 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:48:02.797781 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.798381 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.799345 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:02.799558 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:02.799635 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:02.799928 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:48:02.800419 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:48:02.800529 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:02.800698 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 21:48:02.800750 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:48:02.806715 15710 raft_consensus.cc:1261] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8 [term 1 FOLLOWER]: Refusing update from remote peer 89d4f97e5c28415d9633552e764362a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:48:02.806717 15844 raft_consensus.cc:1261] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc [term 1 FOLLOWER]: Refusing update from remote peer 89d4f97e5c28415d9633552e764362a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:48:02.807436 15918 consensus_queue.cc:1035] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:48:02.807631 15914 consensus_queue.cc:1035] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:48:02.815385 15921 mvcc.cc:205] Tried to move back new op lower bound from 7299039571168440320 to 7299039570864390144. Current Snapshot: MvccSnapshot[applied={T|T < 7299039571168440320}]
I20260620 21:48:02.815577 15922 mvcc.cc:205] Tried to move back new op lower bound from 7299039571168440320 to 7299039570864390144. Current Snapshot: MvccSnapshot[applied={T|T < 7299039571168440320}]
I20260620 21:48:02.819614 15924 mvcc.cc:205] Tried to move back new op lower bound from 7299039571168440320 to 7299039570864390144. Current Snapshot: MvccSnapshot[applied={T|T < 7299039571168440320}]
I20260620 21:48:02.825011 15429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:48:02.825160 15429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:48:02.827576 15429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: Sending DeleteTablet for 3 replicas of tablet 9ed6f9d389e546cea6f5f791c1e4a12c
I20260620 21:48:02.828480 15824 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed6f9d389e546cea6f5f791c1e4a12c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:48:02 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:45736
I20260620 21:48:02.828480 15690 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed6f9d389e546cea6f5f791c1e4a12c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:48:02 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:53718
I20260620 21:48:02.829032 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 15493
I20260620 21:48:02.829020 15931 tablet_replica.cc:330] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc: stopping tablet replica
I20260620 21:48:02.829005 15556 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed6f9d389e546cea6f5f791c1e4a12c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:48:02 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:40582
I20260620 21:48:02.829339 15930 tablet_replica.cc:330] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8: stopping tablet replica
I20260620 21:48:02.829699 15930 raft_consensus.cc:2229] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:48:02.830068 15930 raft_consensus.cc:2258] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:48:02.830121 15931 raft_consensus.cc:2229] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:48:02.830569 15931 raft_consensus.cc:2258] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:48:02.832041 15931 ts_tablet_manager.cc:1905] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:48:02.832448 15930 ts_tablet_manager.cc:1905] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:48:02.834839 15931 ts_tablet_manager.cc:1918] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:48:02.834944 15931 log.cc:1196] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/9ed6f9d389e546cea6f5f791c1e4a12c
I20260620 21:48:02.835307 15931 ts_tablet_manager.cc:1939] T 9ed6f9d389e546cea6f5f791c1e4a12c P e4347d5750a24d21a19dbd71663b08bc: Deleting consensus metadata
I20260620 21:48:02.835391 15930 ts_tablet_manager.cc:1918] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:48:02.835513 15930 log.cc:1196] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/9ed6f9d389e546cea6f5f791c1e4a12c
I20260620 21:48:02.835836 15930 ts_tablet_manager.cc:1939] T 9ed6f9d389e546cea6f5f791c1e4a12c P 44286227758740a0944e5091566231b8: Deleting consensus metadata
I20260620 21:48:02.836764 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 15627
I20260620 21:48:02.836740 15415 catalog_manager.cc:4841] TS e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423): tablet 9ed6f9d389e546cea6f5f791c1e4a12c (table test-table [id=9cdc308fded94770bbb6bdb0733725a9]) successfully deleted
W20260620 21:48:02.837047 15415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.155.129:34437 (error 108)
I20260620 21:48:02.838586 15903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.838247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:34437 (local address 127.0.0.1:40614)
0620 21:48:02.838391 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:02.838472 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:34437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 21:48:02.838778 15415 catalog_manager.cc:4584] TS 89d4f97e5c28415d9633552e764362a5 (127.1.155.129:34437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ed6f9d389e546cea6f5f791c1e4a12c: Network error: Client connection negotiation failed: client connection to 127.1.155.129:34437: connect: Connection refused (error 111)
I20260620 21:48:02.842782 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 15761
I20260620 21:48:02.842967 15903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:02.842680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:34019 (local address 127.0.0.1:53734)
0620 21:48:02.842811 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:02.842876 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.130:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 21:48:02.848483 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 15399
I20260620 21:48:02.854758 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:39637
--webserver_interface=127.1.155.190
--webserver_port=35275
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:39637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:48:02.964422 15933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:02.964730 15933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:02.964802 15933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:02.968848 15933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:48:02.968940 15933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:02.968978 15933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:02.969034 15933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:48:02.969074 15933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:48:02.974328 15933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:39637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:39637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:02.975344 15933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:02.976306 15933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:02.989497 15939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:02.989679 15942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:02.989831 15940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:02.990876 15933 server_base.cc:1034] running on GCE node
I20260620 21:48:02.991587 15933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:02.992790 15933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:02.993985 15933 hybrid_clock.cc:648] HybridClock initialized: now 1781992082993947 us; error 52 us; skew 500 ppm
Jun 20 21:48:02 dist-test-slave-5bvn krb5kdc[15386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992082, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:02.996991 15933 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:48:02.998116 15933 webserver.cc:460] Webserver started at http://127.1.155.190:35275/ using document root <none> and password file <none>
I20260620 21:48:02.998733 15933 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:02.998847 15933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:03.002775 15933 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 21:48:03.004984 15949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:03.006311 15933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:48:03.006453 15933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "de3d7860a515454f80ea87f3cb31b634"
format_stamp: "Formatted at 2026-06-20 21:48:01 on dist-test-slave-5bvn"
server_key: "8da1e54d49ce66ce64dccf4ea60c8806"
server_key_iv: "53a67a1b67ff3be7f37c7892c6f0592c"
server_key_version: "encryptionkey@0"
I20260620 21:48:03.007062 15933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:03.035413 15933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:03.038834 15933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:03.039049 15933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:03.046170 15933 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:39637
I20260620 21:48:03.046470 16001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:39637 every 8 connection(s)
I20260620 21:48:03.047282 15933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:48:03.051273 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 15933
I20260620 21:48:03.051728 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:34437
--local_ip_for_outbound_sockets=127.1.155.129
--tserver_master_addrs=127.1.155.190:39637
--webserver_port=33461
--webserver_interface=127.1.155.129
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:48:03.051813 16002 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:48:03.054133 16002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: Bootstrap starting.
I20260620 21:48:03.071077 16002 log.cc:824] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:03.079792 16002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:48:03.080188 16002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: Bootstrap complete.
I20260620 21:48:03.084270 16002 raft_consensus.cc:348] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } }
I20260620 21:48:03.084645 16002 raft_consensus.cc:729] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3d7860a515454f80ea87f3cb31b634, State: Initialized, Role: FOLLOWER
I20260620 21:48:03.085139 16002 consensus_queue.cc:260] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } }
I20260620 21:48:03.085258 16002 raft_consensus.cc:388] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:48:03.085354 16002 raft_consensus.cc:482] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:48:03.085469 16002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:48:03.087423 16002 raft_consensus.cc:504] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } }
I20260620 21:48:03.087766 16002 leader_election.cc:302] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de3d7860a515454f80ea87f3cb31b634; no voters:
I20260620 21:48:03.088102 16002 leader_election.cc:288] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 21:48:03.088229 16006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:48:03.088470 16006 raft_consensus.cc:686] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 2 LEADER]: Becoming Leader. State: Replica: de3d7860a515454f80ea87f3cb31b634, State: Running, Role: LEADER
I20260620 21:48:03.088793 16006 consensus_queue.cc:237] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } }
I20260620 21:48:03.089326 16002 sys_catalog.cc:564] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:48:03.090155 16007 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "de3d7860a515454f80ea87f3cb31b634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } } }
I20260620 21:48:03.090277 16007 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [sys.catalog]: This master's current role is: LEADER
I20260620 21:48:03.090571 16008 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de3d7860a515454f80ea87f3cb31b634. Latest consensus state: current_term: 2 leader_uuid: "de3d7860a515454f80ea87f3cb31b634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } } }
I20260620 21:48:03.090658 16008 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [sys.catalog]: This master's current role is: LEADER
I20260620 21:48:03.090991 16015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:48:03.095898 16015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:48:03.096632 16015 catalog_manager.cc:1259] Loaded cluster ID: 6abebb26bcb2422e88efde2e9161669a
I20260620 21:48:03.096701 16015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:48:03.098114 16015 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 21:48:03.102608 16015 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 21:48:03.108124 16015 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:48:03.108985 16015 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: Loaded TSK: 0
I20260620 21:48:03.110258 16015 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:48:03.168895 16004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:03.169159 16004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:03.169204 16004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:03.173329 16004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:03.173399 16004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:03.173458 16004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:48:03.178908 16004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:34437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=33461
--tserver_master_addrs=127.1.155.190:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:03.180066 16004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:03.181124 16004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:03.191237 16030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:03.191393 16032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:03.191226 16029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:03.191828 16004 server_base.cc:1034] running on GCE node
I20260620 21:48:03.192303 16004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:03.192981 16004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:03.194156 16004 hybrid_clock.cc:648] HybridClock initialized: now 1781992083194140 us; error 31 us; skew 500 ppm
Jun 20 21:48:03 dist-test-slave-5bvn krb5kdc[15386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992083, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:03.196908 16004 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:48:03.197916 16004 webserver.cc:460] Webserver started at http://127.1.155.129:33461/ using document root <none> and password file <none>
I20260620 21:48:03.198470 16004 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:03.198544 16004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:03.202456 16004 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:48:03.204665 16039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:03.206121 16004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:48:03.206250 16004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "89d4f97e5c28415d9633552e764362a5"
format_stamp: "Formatted at 2026-06-20 21:48:01 on dist-test-slave-5bvn"
server_key: "2c86143ff3fac0b5734504aceb439007"
server_key_iv: "722892b9e377e8877d0e28d173a193cc"
server_key_version: "encryptionkey@0"
I20260620 21:48:03.206876 16004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:03.226734 16004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:03.230173 16004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:03.230382 16004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:03.231055 16004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:48:03.232239 16046 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 21:48:03.236435 16004 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 21:48:03.236505 16004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260620 21:48:03.236554 16004 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 21:48:03.238260 16004 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 21:48:03.238320 16004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 21:48:03.238569 16046 tablet_bootstrap.cc:492] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: Bootstrap starting.
I20260620 21:48:03.249243 16004 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:34437
I20260620 21:48:03.249267 16153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:34437 every 8 connection(s)
I20260620 21:48:03.250488 16004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:48:03.258785 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 16004
I20260620 21:48:03.259047 16046 log.cc:824] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:03.259298 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:34019
--local_ip_for_outbound_sockets=127.1.155.130
--tserver_master_addrs=127.1.155.190:39637
--webserver_port=34547
--webserver_interface=127.1.155.130
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:48:03 dist-test-slave-5bvn krb5kdc[15386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992083, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:48:03.267084 16046 tablet_bootstrap.cc:492] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:48:03.267557 16046 tablet_bootstrap.cc:492] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: Bootstrap complete.
I20260620 21:48:03.268922 16046 ts_tablet_manager.cc:1397] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: Time spent bootstrapping tablet: real 0.031s user 0.027s sys 0.000s
I20260620 21:48:03.269177 16157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.252717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:54945)
0620 21:48:03.253266 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:03.253308 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:03.254217 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:03.255324 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:03.255332 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:03.255757 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:03.256222 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.256237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.257552 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.257555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.258510 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.258520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.259732 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.259738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.259963 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.260678 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:03.260705 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:03.262679 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:03.265503 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:03.265509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:03.265522 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:03.265893 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:03.266354 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:03.266357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:03.266359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:03.266520 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:03.267296 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:03.267303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:03.267628 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 21:48:03.267829 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 21:48:03.268171 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":154,"threads_started":1}
I20260620 21:48:03.269757 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.252709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:54945 (local address 127.1.155.190:39637)
0620 21:48:03.253083 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:03.253103 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:03.253436 (+ 333us) server_negotiation.cc:407] Connection header received
0620 21:48:03.254638 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:03.254648 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:03.254941 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:03.255278 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:03.256411 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.257371 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.258645 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.259299 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.259489 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.262920 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:03.262930 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:03.262950 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:03.262990 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:03.265358 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:03.266033 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:03.266043 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:03.266051 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:03.266226 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:03.266640 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:03.266647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:03.266650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:03.267161 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:03.267320 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:03.268016 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 21:48:03.268366 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":129,"threads_started":1}
I20260620 21:48:03.271683 16046 raft_consensus.cc:348] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } } peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } }
I20260620 21:48:03.272302 16046 raft_consensus.cc:729] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d4f97e5c28415d9633552e764362a5, State: Initialized, Role: FOLLOWER
I20260620 21:48:03.272886 16046 consensus_queue.cc:260] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } } peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } }
I20260620 21:48:03.273398 16154 heartbeater.cc:347] Connected to a master server at 127.1.155.190:39637
I20260620 21:48:03.273622 16154 heartbeater.cc:464] Registering TS with master...
I20260620 21:48:03.273957 16046 ts_tablet_manager.cc:1428] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260620 21:48:03.274169 16154 heartbeater.cc:511] Master 127.1.155.190:39637 requested a full tablet report, sending...
I20260620 21:48:03.276787 15966 ts_manager.cc:194] Registered new tserver with Master: 89d4f97e5c28415d9633552e764362a5 (127.1.155.129:34437)
I20260620 21:48:03.280915 15966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:54945
I20260620 21:48:03.284017 16154 heartbeater.cc:503] Master 127.1.155.190:39637 was elected leader, sending a full tablet report...
Jun 20 21:48:03 dist-test-slave-5bvn krb5kdc[15386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992082, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for kudu/127.1.155.129@KRBTEST.COM
I20260620 21:48:03.290593 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.280211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:34437 (local address 127.0.0.1:40626)
0620 21:48:03.280966 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:03.281028 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:03.281228 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:03.281697 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:03.281702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:03.282136 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:03.282249 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.282255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.283236 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.283240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.283932 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.283941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.284666 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.284669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.284782 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.285329 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:03.285341 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:03.286705 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:03.288694 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:03.288701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:03.288705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:03.289001 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:03.289295 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:03.289297 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:03.289298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:03.289374 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:03.290001 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:03.290008 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:03.290120 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 21:48:03.290240 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 21:48:03.290428 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":123,"threads_started":1}
I20260620 21:48:03.290726 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.280342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40626 (local address 127.1.155.129:34437)
0620 21:48:03.280724 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:03.280736 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:03.281116 (+ 380us) server_negotiation.cc:407] Connection header received
0620 21:48:03.281384 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:03.281394 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:03.281550 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:03.281697 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:03.282369 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.283110 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.284061 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.284545 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.284662 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.286850 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:03.286859 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:03.286871 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:03.286897 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:03.288575 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:03.289104 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:03.289109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:03.289111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:03.289204 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:03.289456 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:03.289459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:03.289461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:03.289760 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:03.289930 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:03.290351 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 21:48:03.290519 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":289,"thread_start_us":100,"threads_started":1}
I20260620 21:48:03.291957 16088 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed6f9d389e546cea6f5f791c1e4a12c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:48:02 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:40626
I20260620 21:48:03.292805 16163 tablet_replica.cc:330] stopping tablet replica
I20260620 21:48:03.293089 16163 raft_consensus.cc:2229] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:48:03.293331 16163 raft_consensus.cc:2258] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:48:03.294354 16163 ts_tablet_manager.cc:1905] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:48:03.297925 16163 ts_tablet_manager.cc:1918] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:48:03.298010 16163 log.cc:1196] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal/wals/9ed6f9d389e546cea6f5f791c1e4a12c
I20260620 21:48:03.298281 16163 ts_tablet_manager.cc:1939] T 9ed6f9d389e546cea6f5f791c1e4a12c P 89d4f97e5c28415d9633552e764362a5: Deleting consensus metadata
I20260620 21:48:03.299887 15950 catalog_manager.cc:4841] TS 89d4f97e5c28415d9633552e764362a5 (127.1.155.129:34437): tablet 9ed6f9d389e546cea6f5f791c1e4a12c (table test-table [id=9cdc308fded94770bbb6bdb0733725a9]) successfully deleted
W20260620 21:48:03.379185 16159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:03.379468 16159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:03.379544 16159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:03.383498 16159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:03.383586 16159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:03.383678 16159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:48:03.388837 16159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:34019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=34547
--tserver_master_addrs=127.1.155.190:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:03.389954 16159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:03.391011 16159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:03.400626 16168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:03.400667 16169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:03.400667 16171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:03.400949 16159 server_base.cc:1034] running on GCE node
I20260620 21:48:03.401405 16159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:03.402092 16159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:03.403263 16159 hybrid_clock.cc:648] HybridClock initialized: now 1781992083403237 us; error 29 us; skew 500 ppm
Jun 20 21:48:03 dist-test-slave-5bvn krb5kdc[15386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992083, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:03.406131 16159 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:48:03.406924 16159 webserver.cc:460] Webserver started at http://127.1.155.130:34547/ using document root <none> and password file <none>
I20260620 21:48:03.407416 16159 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:03.407485 16159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:03.411303 16159 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:48:03.413488 16178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:03.414831 16159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:48:03.414978 16159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "44286227758740a0944e5091566231b8"
format_stamp: "Formatted at 2026-06-20 21:48:02 on dist-test-slave-5bvn"
server_key: "fab0e3d7bff612e7b4793dda3633cfda"
server_key_iv: "e0f59244ff22b66788ec5293c5a6dd25"
server_key_version: "encryptionkey@0"
I20260620 21:48:03.415565 16159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:03.429147 16159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:03.432665 16159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:03.432889 16159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:03.433502 16159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:48:03.434381 16159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:48:03.434453 16159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:03.434530 16159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:48:03.434579 16159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:03.443931 16159 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:34019
I20260620 21:48:03.443949 16291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:34019 every 8 connection(s)
I20260620 21:48:03.445014 16159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:48:03.446182 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 16159
I20260620 21:48:03.446614 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:45423
--local_ip_for_outbound_sockets=127.1.155.131
--tserver_master_addrs=127.1.155.190:39637
--webserver_port=37519
--webserver_interface=127.1.155.131
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:48:03 dist-test-slave-5bvn krb5kdc[15386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992083, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:48:03.459870 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.447222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:54385 (local address 127.1.155.190:39637)
0620 21:48:03.447395 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:03.447400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:03.448368 (+ 968us) server_negotiation.cc:407] Connection header received
0620 21:48:03.449679 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:03.449691 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:03.449763 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:03.449903 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:03.451084 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.452014 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.453184 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.453626 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.453762 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.456230 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:03.456235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:03.456239 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:03.456271 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:03.457922 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:03.458417 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:03.458421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:03.458423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:03.458504 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:03.458809 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:03.458812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:03.458814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:03.459017 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:03.459109 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:03.459554 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 21:48:03.459681 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 21:48:03.460628 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.447670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:54385)
0620 21:48:03.448157 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:03.448249 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:03.449463 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:03.450063 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:03.450069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:03.450453 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:03.450924 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.450937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.452160 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.452163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.453058 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.453067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.453782 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.453785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.454009 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.454616 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:03.454633 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:03.456056 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:03.458046 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:03.458051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:03.458063 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:03.458325 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:03.458587 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:03.458589 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:03.458591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:03.458727 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:03.459109 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:03.459113 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:03.459340 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 21:48:03.459495 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 21:48:03.459763 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":149,"threads_started":1}
I20260620 21:48:03.461870 16292 heartbeater.cc:347] Connected to a master server at 127.1.155.190:39637
I20260620 21:48:03.462123 16292 heartbeater.cc:464] Registering TS with master...
I20260620 21:48:03.462604 16292 heartbeater.cc:511] Master 127.1.155.190:39637 requested a full tablet report, sending...
I20260620 21:48:03.463626 15966 ts_manager.cc:194] Registered new tserver with Master: 44286227758740a0944e5091566231b8 (127.1.155.130:34019)
I20260620 21:48:03.464346 15966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:54385
W20260620 21:48:03.559450 16294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:03.559700 16294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:03.559742 16294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:03.563700 16294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:03.563752 16294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:03.563808 16294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:48:03.568984 16294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:45423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=37519
--tserver_master_addrs=127.1.155.190:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:03.570012 16294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:03.570950 16294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:03.580185 16301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:03.580178 16303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:03.580173 16300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:03.580475 16294 server_base.cc:1034] running on GCE node
I20260620 21:48:03.580960 16294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:03.581607 16294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:03.582800 16294 hybrid_clock.cc:648] HybridClock initialized: now 1781992083582773 us; error 39 us; skew 500 ppm
Jun 20 21:48:03 dist-test-slave-5bvn krb5kdc[15386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992083, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:03.585585 16294 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:48:03.586511 16294 webserver.cc:460] Webserver started at http://127.1.155.131:37519/ using document root <none> and password file <none>
I20260620 21:48:03.587069 16294 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:03.587177 16294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:03.591006 16294 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 21:48:03.593041 16310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:03.594368 16294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:48:03.594507 16294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "e4347d5750a24d21a19dbd71663b08bc"
format_stamp: "Formatted at 2026-06-20 21:48:02 on dist-test-slave-5bvn"
server_key: "7874a4db27361a231224b9ab50efe95c"
server_key_iv: "2343f9ca95956857af8ee604dd95336e"
server_key_version: "encryptionkey@0"
I20260620 21:48:03.595044 16294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:03.604256 16294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:03.607551 16294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:03.607748 16294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:03.608310 16294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:48:03.609156 16294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:48:03.609229 16294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:03.609300 16294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:48:03.609354 16294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:03.619100 16294 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:45423
I20260620 21:48:03.619122 16423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:45423 every 8 connection(s)
I20260620 21:48:03.620216 16294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:48:03.622980 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 16294
Jun 20 21:48:03 dist-test-slave-5bvn krb5kdc[15386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992083, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:48:03.632579 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.623701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.1.155.190:39637)
0620 21:48:03.623948 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:03.623953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:03.624032 (+ 79us) server_negotiation.cc:407] Connection header received
0620 21:48:03.624132 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:03.624135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:03.624184 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:03.624290 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:03.625152 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.625966 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.626905 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.627175 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.627296 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.628827 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:03.628831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:03.628833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:03.628865 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:03.630529 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:03.631003 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:03.631009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:03.631016 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:03.631119 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:03.631435 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:03.631438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:03.631439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:03.631698 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:03.631913 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:03.632178 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 21:48:03.632355 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
I20260620 21:48:03.634967 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.622049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:40841 (local address 127.1.155.190:39637)
0620 21:48:03.622188 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:03.622191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:03.623080 (+ 889us) server_negotiation.cc:407] Connection header received
0620 21:48:03.624714 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:03.624718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:03.624764 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:03.624839 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:03.626279 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.626760 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.628002 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.628242 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.628357 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.631126 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:03.631129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:03.631131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:03.631157 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:03.632801 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:03.633338 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:03.633340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:03.633342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:03.633414 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:03.633792 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:03.633795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:03.633797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:03.634013 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:03.634121 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:03.634696 (+ 575us) server_negotiation.cc:299] Negotiation successful
0620 21:48:03.634804 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:48:03.635932 16426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.622386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:40841)
0620 21:48:03.622880 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:03.622937 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:03.624522 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:03.625005 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:03.625012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:03.625482 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:03.626151 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.626166 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.626890 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.626894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.627890 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.627901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.628411 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.628415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.628580 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.629134 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:03.629161 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:03.630981 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:03.632924 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:03.632930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:03.632946 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:03.633230 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:03.633533 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:03.633535 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:03.633537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:03.633680 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:03.634122 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:03.634129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:03.634413 (+ 284us) client_negotiation.cc:769] Sending connection context
0620 21:48:03.634639 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 21:48:03.634963 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":111,"threads_started":1}
I20260620 21:48:03.637036 16424 heartbeater.cc:347] Connected to a master server at 127.1.155.190:39637
I20260620 21:48:03.637313 16424 heartbeater.cc:464] Registering TS with master...
I20260620 21:48:03.637830 16424 heartbeater.cc:511] Master 127.1.155.190:39637 requested a full tablet report, sending...
I20260620 21:48:03.638947 15966 ts_manager.cc:194] Registered new tserver with Master: e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423)
I20260620 21:48:03.639510 15966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:40841
I20260620 21:48:03.646519 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:48:03.650736 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.647162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33438 (local address 127.1.155.190:39637)
0620 21:48:03.647327 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:03.647331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:03.647446 (+ 115us) server_negotiation.cc:407] Connection header received
0620 21:48:03.647582 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:03.647584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:03.647676 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:03.647760 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:48:03.647982 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.648475 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.649309 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.649521 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.649594 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.649929 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:48:03.650298 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:48:03.650388 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:03.650517 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 21:48:03.650560 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 21:48:03.651373 15966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:48:03.654561 15966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:48:03.662678 16088 tablet_service.cc:1467] Processing CreateTablet for tablet 2b378879a5b84352b926970d341d26b1 (DEFAULT_TABLE table=test-table [id=1efb8887f1664cf488de94c9320df1ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:48:03.663342 16088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b378879a5b84352b926970d341d26b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:48:03.666201 16046 tablet_bootstrap.cc:492] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5: Bootstrap starting.
I20260620 21:48:03.666787 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.661441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:34019 (local address 127.0.0.1:53750)
0620 21:48:03.661599 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:03.661615 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:03.661697 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:03.662614 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:03.662618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:03.662634 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:03.662762 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.662769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.664154 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.664161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.665136 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.665146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.666362 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.666366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.666465 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.666517 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 21:48:03.666575 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 21:48:03.666658 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 21:48:03.667135 16046 tablet_bootstrap.cc:654] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5: Neither blocks nor log segments found. Creating new log.
I20260620 21:48:03.667129 16434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.661651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:45423 (local address 127.0.0.1:45744)
0620 21:48:03.661927 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:03.661941 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:03.662027 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:03.662801 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:03.662804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:03.662822 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:03.663128 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.663138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.664754 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.664758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.665683 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.665693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.666694 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.666697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.666795 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.666816 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:48:03.666903 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 21:48:03.666972 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":124,"threads_started":1}
I20260620 21:48:03.667505 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.661917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45744 (local address 127.1.155.131:45423)
0620 21:48:03.662344 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:03.662352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:03.662371 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:48:03.662430 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:03.662433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:03.662604 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:03.662747 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:03.663258 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.664602 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.665819 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.666571 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.666693 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.666833 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:03.667278 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 21:48:03.667389 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":130,"threads_started":1}
I20260620 21:48:03.667886 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.661581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.1.155.130:34019)
0620 21:48:03.662102 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:03.662114 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:03.662136 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:48:03.662235 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:03.662240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:03.662453 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:03.662615 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:03.663014 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.664022 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.665285 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.666148 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.667394 (+ 1246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.667573 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:03.667628 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 21:48:03.667716 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":172,"threads_started":1}
I20260620 21:48:03.669193 16046 tablet_bootstrap.cc:492] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5: No bootstrap required, opened a new log
I20260620 21:48:03.669282 16046 ts_tablet_manager.cc:1397] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:48:03.669781 16046 raft_consensus.cc:348] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:03.669956 16046 raft_consensus.cc:374] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:48:03.670006 16046 raft_consensus.cc:729] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d4f97e5c28415d9633552e764362a5, State: Initialized, Role: FOLLOWER
I20260620 21:48:03.670125 16046 consensus_queue.cc:260] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:03.670491 16046 ts_tablet_manager.cc:1428] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 21:48:03.670418 16358 tablet_service.cc:1467] Processing CreateTablet for tablet 2b378879a5b84352b926970d341d26b1 (DEFAULT_TABLE table=test-table [id=1efb8887f1664cf488de94c9320df1ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:48:03.670416 16226 tablet_service.cc:1467] Processing CreateTablet for tablet 2b378879a5b84352b926970d341d26b1 (DEFAULT_TABLE table=test-table [id=1efb8887f1664cf488de94c9320df1ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:48:03.671501 16358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b378879a5b84352b926970d341d26b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:48:03.671497 16226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b378879a5b84352b926970d341d26b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:48:03.679498 16438 tablet_bootstrap.cc:492] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8: Bootstrap starting.
I20260620 21:48:03.680392 16437 tablet_bootstrap.cc:492] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc: Bootstrap starting.
I20260620 21:48:03.682364 16438 tablet_bootstrap.cc:654] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8: Neither blocks nor log segments found. Creating new log.
I20260620 21:48:03.682941 16437 tablet_bootstrap.cc:654] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc: Neither blocks nor log segments found. Creating new log.
I20260620 21:48:03.683374 16438 log.cc:824] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:03.683714 16437 log.cc:824] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:03.685212 16438 tablet_bootstrap.cc:492] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8: No bootstrap required, opened a new log
I20260620 21:48:03.685353 16437 tablet_bootstrap.cc:492] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc: No bootstrap required, opened a new log
I20260620 21:48:03.685448 16438 ts_tablet_manager.cc:1397] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 21:48:03.685572 16437 ts_tablet_manager.cc:1397] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:48:03.688354 16437 raft_consensus.cc:348] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:03.688354 16438 raft_consensus.cc:348] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:03.688580 16437 raft_consensus.cc:374] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:48:03.688608 16438 raft_consensus.cc:374] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:48:03.688614 16437 raft_consensus.cc:729] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4347d5750a24d21a19dbd71663b08bc, State: Initialized, Role: FOLLOWER
I20260620 21:48:03.688673 16438 raft_consensus.cc:729] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44286227758740a0944e5091566231b8, State: Initialized, Role: FOLLOWER
I20260620 21:48:03.689141 16438 consensus_queue.cc:260] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:03.689141 16437 consensus_queue.cc:260] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:03.689819 16424 heartbeater.cc:503] Master 127.1.155.190:39637 was elected leader, sending a full tablet report...
I20260620 21:48:03.689819 16292 heartbeater.cc:503] Master 127.1.155.190:39637 was elected leader, sending a full tablet report...
I20260620 21:48:03.690260 16437 ts_tablet_manager.cc:1428] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 21:48:03.690428 16438 ts_tablet_manager.cc:1428] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260620 21:48:03.695973 16293 tablet.cc:2369] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:48:03.751801 16155 tablet.cc:2369] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:48:03.868054 16442 raft_consensus.cc:482] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:48:03.868259 16442 raft_consensus.cc:504] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:03.869381 16442 leader_election.cc:288] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44286227758740a0944e5091566231b8 (127.1.155.130:34019), 89d4f97e5c28415d9633552e764362a5 (127.1.155.129:34437)
W20260620 21:48:03.871143 16425 tablet.cc:2369] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:48:03.874804 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.869776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:47549 (local address 127.1.155.129:34437)
0620 21:48:03.870106 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:03.870109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:03.870123 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:48:03.870171 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:03.870173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:03.870268 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:03.870392 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:03.870869 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.871643 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.873492 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.874168 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.874277 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.874410 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:03.874614 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 21:48:03.874685 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":110,"threads_started":1}
I20260620 21:48:03.874796 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.869715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:48643 (local address 127.1.155.130:34019)
0620 21:48:03.869870 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:03.869876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:03.870235 (+ 359us) server_negotiation.cc:407] Connection header received
0620 21:48:03.870297 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:03.870300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:03.870360 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:03.870446 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:03.870869 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.871656 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.873492 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.874102 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.874241 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.874293 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:03.874596 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 21:48:03.874674 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:48:03.874804 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.869659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:34019 (local address 127.1.155.131:48643)
0620 21:48:03.870129 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:03.870140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:03.870216 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:03.870450 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:03.870452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:03.870463 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:03.870741 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.870750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.871765 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.871769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.873340 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.873351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.874233 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.874239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.874423 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.874466 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 21:48:03.874550 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 21:48:03.874633 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":321,"thread_start_us":122,"threads_started":1}
I20260620 21:48:03.874794 16426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.869657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:34437 (local address 127.1.155.131:47549)
0620 21:48:03.869777 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:03.869793 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:03.869939 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:03.870430 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:03.870435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:03.870453 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:03.870746 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.870751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.871765 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.871769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.873340 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:03.873351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.874269 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.874272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:03.874483 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.874512 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 21:48:03.874590 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 21:48:03.874641 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":370}
I20260620 21:48:03.875507 16246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b378879a5b84352b926970d341d26b1" candidate_uuid: "e4347d5750a24d21a19dbd71663b08bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44286227758740a0944e5091566231b8" is_pre_election: true
I20260620 21:48:03.875708 16108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b378879a5b84352b926970d341d26b1" candidate_uuid: "e4347d5750a24d21a19dbd71663b08bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d4f97e5c28415d9633552e764362a5" is_pre_election: true
I20260620 21:48:03.875818 16246 raft_consensus.cc:2454] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4347d5750a24d21a19dbd71663b08bc in term 0.
I20260620 21:48:03.875989 16108 raft_consensus.cc:2454] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4347d5750a24d21a19dbd71663b08bc in term 0.
I20260620 21:48:03.876331 16313 leader_election.cc:302] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44286227758740a0944e5091566231b8, e4347d5750a24d21a19dbd71663b08bc; no voters:
I20260620 21:48:03.876616 16442 raft_consensus.cc:2781] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:48:03.876699 16442 raft_consensus.cc:482] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:48:03.876742 16442 raft_consensus.cc:3037] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:48:03.877705 16442 raft_consensus.cc:504] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:03.878113 16442 leader_election.cc:288] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [CANDIDATE]: Term 1 election: Requested vote from peers 44286227758740a0944e5091566231b8 (127.1.155.130:34019), 89d4f97e5c28415d9633552e764362a5 (127.1.155.129:34437)
I20260620 21:48:03.878465 16246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b378879a5b84352b926970d341d26b1" candidate_uuid: "e4347d5750a24d21a19dbd71663b08bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44286227758740a0944e5091566231b8"
I20260620 21:48:03.878509 16108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b378879a5b84352b926970d341d26b1" candidate_uuid: "e4347d5750a24d21a19dbd71663b08bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d4f97e5c28415d9633552e764362a5"
I20260620 21:48:03.878572 16246 raft_consensus.cc:3037] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:48:03.878648 16108 raft_consensus.cc:3037] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:48:03.879588 16246 raft_consensus.cc:2454] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4347d5750a24d21a19dbd71663b08bc in term 1.
I20260620 21:48:03.879634 16108 raft_consensus.cc:2454] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4347d5750a24d21a19dbd71663b08bc in term 1.
I20260620 21:48:03.879848 16313 leader_election.cc:302] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44286227758740a0944e5091566231b8, e4347d5750a24d21a19dbd71663b08bc; no voters:
I20260620 21:48:03.880036 16442 raft_consensus.cc:2781] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:48:03.880259 16442 raft_consensus.cc:686] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [term 1 LEADER]: Becoming Leader. State: Replica: e4347d5750a24d21a19dbd71663b08bc, State: Running, Role: LEADER
I20260620 21:48:03.880568 16442 consensus_queue.cc:237] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:03.883762 15965 catalog_manager.cc:5495] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc reported cstate change: term changed from 0 to 1, leader changed from <none> to e4347d5750a24d21a19dbd71663b08bc (127.1.155.131). New cstate: current_term: 1 leader_uuid: "e4347d5750a24d21a19dbd71663b08bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } health_report { overall_health: UNKNOWN } } }
I20260620 21:48:03.903177 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.899533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45746 (local address 127.1.155.131:45423)
0620 21:48:03.899670 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:03.899677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:03.899693 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:48:03.899879 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:03.899887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:03.899942 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:03.900051 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:48:03.900281 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.900808 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.901649 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:03.901950 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:03.902062 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:03.902371 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:48:03.902754 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:48:03.902892 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:03.902957 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 21:48:03.903012 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:48:03.908902 16108 raft_consensus.cc:1261] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Refusing update from remote peer e4347d5750a24d21a19dbd71663b08bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:48:03.909093 16246 raft_consensus.cc:1261] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8 [term 1 FOLLOWER]: Refusing update from remote peer e4347d5750a24d21a19dbd71663b08bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:48:03.909603 16445 consensus_queue.cc:1035] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:48:03.909865 16442 consensus_queue.cc:1035] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:48:03.916208 16448 mvcc.cc:205] Tried to move back new op lower bound from 7299039575683428352 to 7299039575576809472. Current Snapshot: MvccSnapshot[applied={T|T < 7299039575683428352}]
I20260620 21:48:03.916914 16449 mvcc.cc:205] Tried to move back new op lower bound from 7299039575683428352 to 7299039575576809472. Current Snapshot: MvccSnapshot[applied={T|T < 7299039575683428352}]
I20260620 21:48:03.919303 16450 mvcc.cc:205] Tried to move back new op lower bound from 7299039575683428352 to 7299039575576809472. Current Snapshot: MvccSnapshot[applied={T|T < 7299039575683428352}]
I20260620 21:48:03.924731 15966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:48:03.924906 15966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:48:03.927304 15966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: Sending DeleteTablet for 3 replicas of tablet 2b378879a5b84352b926970d341d26b1
I20260620 21:48:03.928086 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 16004
I20260620 21:48:03.928071 16358 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b378879a5b84352b926970d341d26b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:48:03 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:45744
I20260620 21:48:03.928222 16226 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b378879a5b84352b926970d341d26b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:48:03 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:53750
I20260620 21:48:03.928717 16458 tablet_replica.cc:330] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8: stopping tablet replica
I20260620 21:48:03.929042 16457 tablet_replica.cc:330] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc: stopping tablet replica
I20260620 21:48:03.929032 16458 raft_consensus.cc:2229] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:48:03.929343 16458 raft_consensus.cc:2258] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:48:03.929378 16457 raft_consensus.cc:2229] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:48:03.929951 16457 raft_consensus.cc:2258] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:48:03.931365 16458 ts_tablet_manager.cc:1905] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:48:03.931365 16457 ts_tablet_manager.cc:1905] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:48:03.934072 16457 ts_tablet_manager.cc:1918] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:48:03.934072 16458 ts_tablet_manager.cc:1918] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:48:03.934177 16458 log.cc:1196] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/2b378879a5b84352b926970d341d26b1
I20260620 21:48:03.934307 16457 log.cc:1196] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/2b378879a5b84352b926970d341d26b1
I20260620 21:48:03.934561 16458 ts_tablet_manager.cc:1939] T 2b378879a5b84352b926970d341d26b1 P 44286227758740a0944e5091566231b8: Deleting consensus metadata
I20260620 21:48:03.934809 16457 ts_tablet_manager.cc:1939] T 2b378879a5b84352b926970d341d26b1 P e4347d5750a24d21a19dbd71663b08bc: Deleting consensus metadata
I20260620 21:48:03.935987 15950 catalog_manager.cc:4841] TS e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423): tablet 2b378879a5b84352b926970d341d26b1 (table test-table [id=1efb8887f1664cf488de94c9320df1ab]) successfully deleted
I20260620 21:48:03.935807 15952 catalog_manager.cc:4841] TS 44286227758740a0944e5091566231b8 (127.1.155.130:34019): tablet 2b378879a5b84352b926970d341d26b1 (table test-table [id=1efb8887f1664cf488de94c9320df1ab]) successfully deleted
W20260620 21:48:03.937078 15950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.155.129:34437 (error 108)
I20260620 21:48:03.937561 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 16159
I20260620 21:48:03.938700 16434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:03.938439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:34437 (local address 127.0.0.1:40640)
0620 21:48:03.938562 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:03.938632 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:34437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 21:48:03.938921 15950 catalog_manager.cc:4584] TS 89d4f97e5c28415d9633552e764362a5 (127.1.155.129:34437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b378879a5b84352b926970d341d26b1: Network error: Client connection negotiation failed: client connection to 127.1.155.129:34437: connect: Connection refused (error 111)
I20260620 21:48:03.945156 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 16294
I20260620 21:48:03.949919 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 15933
I20260620 21:48:03.955947 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:39637
--webserver_interface=127.1.155.190
--webserver_port=35275
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:39637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:48:04.063999 16460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:04.064320 16460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:04.064406 16460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:04.068509 16460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:48:04.068599 16460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:04.068637 16460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:04.068694 16460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:48:04.068732 16460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:48:04.074008 16460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:39637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:39637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:04.075017 16460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:04.075987 16460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:04.084345 16465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:04.084558 16460 server_base.cc:1034] running on GCE node
W20260620 21:48:04.084692 16468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:04.085424 16466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:04.086077 16460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:04.087111 16460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:04.088274 16460 hybrid_clock.cc:648] HybridClock initialized: now 1781992084088259 us; error 26 us; skew 500 ppm
Jun 20 21:48:04 dist-test-slave-5bvn krb5kdc[15386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:04.091004 16460 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:48:04.091843 16460 webserver.cc:460] Webserver started at http://127.1.155.190:35275/ using document root <none> and password file <none>
I20260620 21:48:04.092337 16460 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:04.092392 16460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:04.096040 16460 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 21:48:04.098098 16475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:04.099255 16460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:48:04.099350 16460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "de3d7860a515454f80ea87f3cb31b634"
format_stamp: "Formatted at 2026-06-20 21:48:01 on dist-test-slave-5bvn"
server_key: "8da1e54d49ce66ce64dccf4ea60c8806"
server_key_iv: "53a67a1b67ff3be7f37c7892c6f0592c"
server_key_version: "encryptionkey@0"
I20260620 21:48:04.099895 16460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:04.120214 16460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:04.123533 16460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:04.123740 16460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:04.130856 16460 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:39637
I20260620 21:48:04.131153 16527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:39637 every 8 connection(s)
I20260620 21:48:04.131948 16460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:48:04.136349 16528 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:48:04.138926 16528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: Bootstrap starting.
I20260620 21:48:04.141984 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 16460
I20260620 21:48:04.142443 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:34437
--local_ip_for_outbound_sockets=127.1.155.129
--tserver_master_addrs=127.1.155.190:39637
--webserver_port=33461
--webserver_interface=127.1.155.129
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:48:04.157752 16528 log.cc:824] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:04.168917 16528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:48:04.169292 16528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634: Bootstrap complete.
I20260620 21:48:04.173178 16528 raft_consensus.cc:348] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } }
I20260620 21:48:04.173554 16528 raft_consensus.cc:729] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3d7860a515454f80ea87f3cb31b634, State: Initialized, Role: FOLLOWER
I20260620 21:48:04.174096 16528 consensus_queue.cc:260] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } }
I20260620 21:48:04.174216 16528 raft_consensus.cc:388] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:48:04.174304 16528 raft_consensus.cc:482] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:48:04.174423 16528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 2 FOLLOWER]: Advancing to term 3
I20260620 21:48:04.176205 16528 raft_consensus.cc:504] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } }
I20260620 21:48:04.176537 16528 leader_election.cc:302] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de3d7860a515454f80ea87f3cb31b634; no voters:
I20260620 21:48:04.176930 16528 leader_election.cc:288] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 21:48:04.177013 16532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 21:48:04.177198 16532 raft_consensus.cc:686] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [term 3 LEADER]: Becoming Leader. State: Replica: de3d7860a515454f80ea87f3cb31b634, State: Running, Role: LEADER
I20260620 21:48:04.177474 16532 consensus_queue.cc:237] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } }
I20260620 21:48:04.178175 16528 sys_catalog.cc:564] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:48:04.178411 16533 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de3d7860a515454f80ea87f3cb31b634. Latest consensus state: current_term: 3 leader_uuid: "de3d7860a515454f80ea87f3cb31b634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } } }
I20260620 21:48:04.178442 16534 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "de3d7860a515454f80ea87f3cb31b634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3d7860a515454f80ea87f3cb31b634" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39637 } } }
I20260620 21:48:04.178534 16533 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [sys.catalog]: This master's current role is: LEADER
I20260620 21:48:04.178548 16534 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3d7860a515454f80ea87f3cb31b634 [sys.catalog]: This master's current role is: LEADER
I20260620 21:48:04.178854 16536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:48:04.183983 16536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:48:04.184643 16536 catalog_manager.cc:1259] Loaded cluster ID: 6abebb26bcb2422e88efde2e9161669a
I20260620 21:48:04.184710 16536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 21:48:04.185709 16536 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4cc4e099ad google::LogMessage::Fail() at ??:0
@ 0x7f4cc4e0c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4cc4e094de google::LogMessage::Flush() at ??:0
@ 0x7f4cc4e0aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4ccf748fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4ccf749d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4ccf746579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4ccf78a640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4cc705b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4cc705bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4cc705d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4cc704c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4cc86906db start_thread at ??:0
@ 0x7f4cc390761f clone at ??:0
W20260620 21:48:04.277420 16530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:04.277657 16530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:04.277699 16530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:04.281705 16530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:04.281756 16530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:04.281840 16530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:48:04.286955 16530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:34437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=33461
--tserver_master_addrs=127.1.155.190:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:04.287958 16530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:04.288924 16530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:04.298095 16572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:04.298120 16570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:04.298113 16569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:04.298627 16530 server_base.cc:1034] running on GCE node
I20260620 21:48:04.299060 16530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:04.299782 16530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:04.300984 16530 hybrid_clock.cc:648] HybridClock initialized: now 1781992084300945 us; error 47 us; skew 500 ppm
Jun 20 21:48:04 dist-test-slave-5bvn krb5kdc[15386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:04.304462 16530 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:48:04.305532 16530 webserver.cc:460] Webserver started at http://127.1.155.129:33461/ using document root <none> and password file <none>
I20260620 21:48:04.306241 16530 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:04.306388 16530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:04.310753 16530 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 21:48:04.313234 16579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:04.314599 16530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:48:04.314735 16530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "89d4f97e5c28415d9633552e764362a5"
format_stamp: "Formatted at 2026-06-20 21:48:01 on dist-test-slave-5bvn"
server_key: "2c86143ff3fac0b5734504aceb439007"
server_key_iv: "722892b9e377e8877d0e28d173a193cc"
server_key_version: "encryptionkey@0"
I20260620 21:48:04.315274 16530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:04.330253 16530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:04.333420 16530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:04.333628 16530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:04.334260 16530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:48:04.335316 16586 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 21:48:04.340608 16530 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 21:48:04.340706 16530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 21:48:04.340799 16530 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 21:48:04.342674 16530 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 21:48:04.342752 16530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 21:48:04.343024 16586 tablet_bootstrap.cc:492] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5: Bootstrap starting.
I20260620 21:48:04.354746 16530 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:34437
I20260620 21:48:04.355012 16693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:34437 every 8 connection(s)
I20260620 21:48:04.355859 16530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:48:04.359567 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 16530
I20260620 21:48:04.360097 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:34019
--local_ip_for_outbound_sockets=127.1.155.130
--tserver_master_addrs=127.1.155.190:39637
--webserver_port=34547
--webserver_interface=127.1.155.130
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:48:04.364172 16586 log.cc:824] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:04.372738 16586 tablet_bootstrap.cc:492] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:48:04.373152 16586 tablet_bootstrap.cc:492] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5: Bootstrap complete.
I20260620 21:48:04.374583 16586 ts_tablet_manager.cc:1397] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.012s
I20260620 21:48:04.377871 16586 raft_consensus.cc:348] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:04.378432 16586 raft_consensus.cc:729] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d4f97e5c28415d9633552e764362a5, State: Initialized, Role: FOLLOWER
I20260620 21:48:04.378952 16586 consensus_queue.cc:260] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:04.380014 16586 ts_tablet_manager.cc:1428] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 21:48:04.393489 16696 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:48:04.358201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:32907)
0620 21:48:04.358691 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:04.358745 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:04.359868 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:04.392768 (+ 32900us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":155,"threads_started":1}
W20260620 21:48:04.394074 16694 heartbeater.cc:644] Failed to heartbeat to 127.1.155.190:39637 (0 consecutive failures): Network error: Failed to ping master at 127.1.155.190:39637: Client connection negotiation failed: client connection to 127.1.155.190:39637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 21:48:04.394835 16696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:04.394584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:51211)
0620 21:48:04.394719 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:04.394780 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 21:48:04.395490 16696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:04.395257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:50201)
0620 21:48:04.395365 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:04.395445 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 21:48:04.395654 16694 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:48:04.477070 16697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:04.477371 16697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:04.477455 16697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:04.481623 16697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:04.481830 16697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:04.482005 16697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:48:04.487802 16697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:34019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=34547
--tserver_master_addrs=127.1.155.190:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:04.488840 16697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:04.489853 16697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:48:04.499171 16697 server_base.cc:1034] running on GCE node
W20260620 21:48:04.499120 16707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:04.499094 16704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:04.499099 16705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:04.499645 16697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:04.500224 16697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:04.501386 16697 hybrid_clock.cc:648] HybridClock initialized: now 1781992084501372 us; error 25 us; skew 500 ppm
Jun 20 21:48:04 dist-test-slave-5bvn krb5kdc[15386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:04.504099 16697 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:48:04.504927 16697 webserver.cc:460] Webserver started at http://127.1.155.130:34547/ using document root <none> and password file <none>
I20260620 21:48:04.505455 16697 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:04.505544 16697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:04.509076 16697 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 21:48:04.511067 16714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:04.512310 16697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:48:04.512454 16697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "44286227758740a0944e5091566231b8"
format_stamp: "Formatted at 2026-06-20 21:48:02 on dist-test-slave-5bvn"
server_key: "fab0e3d7bff612e7b4793dda3633cfda"
server_key_iv: "e0f59244ff22b66788ec5293c5a6dd25"
server_key_version: "encryptionkey@0"
I20260620 21:48:04.513026 16697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:04.536109 16697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:04.539543 16697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:04.539755 16697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:04.540360 16697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:48:04.541178 16697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:48:04.541251 16697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:04.541322 16697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:48:04.541365 16697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:04.551287 16697 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:34019
I20260620 21:48:04.551307 16827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:34019 every 8 connection(s)
I20260620 21:48:04.552268 16697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:48:04.556107 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:04.554405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:57731)
0620 21:48:04.554874 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:04.554980 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":105,"threads_started":1}
W20260620 21:48:04.556581 16828 heartbeater.cc:644] Failed to heartbeat to 127.1.155.190:39637 (0 consecutive failures): Network error: Failed to ping master at 127.1.155.190:39637: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
I20260620 21:48:04.557107 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:04.556911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:32895)
0620 21:48:04.557010 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:04.557057 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 21:48:04.557689 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:04.557515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:36749)
0620 21:48:04.557606 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:04.557645 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 21:48:04.557899 16828 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 21:48:04.558753 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 16697
I20260620 21:48:04.559163 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:45423
--local_ip_for_outbound_sockets=127.1.155.131
--tserver_master_addrs=127.1.155.190:39637
--webserver_port=37519
--webserver_interface=127.1.155.131
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:48:04.677062 16831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:04.677311 16831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:04.677352 16831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:04.681288 16831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:04.681344 16831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:04.681402 16831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:48:04.686506 16831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:45423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=37519
--tserver_master_addrs=127.1.155.190:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:04.687479 16831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:04.688414 16831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:04.698105 16839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:04.698100 16837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:04.698338 16831 server_base.cc:1034] running on GCE node
W20260620 21:48:04.698104 16836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:04.698778 16831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:04.699409 16831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:04.700579 16831 hybrid_clock.cc:648] HybridClock initialized: now 1781992084700547 us; error 42 us; skew 500 ppm
Jun 20 21:48:04 dist-test-slave-5bvn krb5kdc[15386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:04.703467 16831 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:48:04.704350 16831 webserver.cc:460] Webserver started at http://127.1.155.131:37519/ using document root <none> and password file <none>
I20260620 21:48:04.704922 16831 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:04.704982 16831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:04.708949 16831 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:48:04.711112 16846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:04.712409 16831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:48:04.712549 16831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "e4347d5750a24d21a19dbd71663b08bc"
format_stamp: "Formatted at 2026-06-20 21:48:02 on dist-test-slave-5bvn"
server_key: "7874a4db27361a231224b9ab50efe95c"
server_key_iv: "2343f9ca95956857af8ee604dd95336e"
server_key_version: "encryptionkey@0"
I20260620 21:48:04.713092 16831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:04.746400 16831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:04.749365 16831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:04.749581 16831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:04.750197 16831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:48:04.751029 16831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:48:04.751101 16831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:04.751168 16831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:48:04.751220 16831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:04.760244 16831 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:45423
I20260620 21:48:04.760311 16959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:45423 every 8 connection(s)
I20260620 21:48:04.761256 16831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:48:04.764824 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:04.763406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:57411)
0620 21:48:04.764046 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:04.764149 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":130,"threads_started":1}
W20260620 21:48:04.765311 16960 heartbeater.cc:644] Failed to heartbeat to 127.1.155.190:39637 (0 consecutive failures): Network error: Failed to ping master at 127.1.155.190:39637: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
I20260620 21:48:04.765812 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 16831
I20260620 21:48:04.765945 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:04.765664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:42929)
0620 21:48:04.765788 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:04.765874 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 21:48:04.766600 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:04.766344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:48371)
0620 21:48:04.766463 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:04.766526 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 21:48:04.766845 16960 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 21:48:05.396752 16964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:05.396236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:51297)
0620 21:48:05.396498 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:05.396595 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
I20260620 21:48:05.558789 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:05.558400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:44705)
0620 21:48:05.558633 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:05.558690 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260620 21:48:05.767740 16966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:05.767358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:50257)
0620 21:48:05.767593 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:05.767648 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":66,"threads_started":1}
I20260620 21:48:06.207064 16967 raft_consensus.cc:482] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:48:06.207268 16967 raft_consensus.cc:504] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:06.208401 16967 leader_election.cc:288] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44286227758740a0944e5091566231b8 (127.1.155.130:34019), e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423)
Jun 20 21:48:06 dist-test-slave-5bvn krb5kdc[15386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.131@KRBTEST.COM
Jun 20 21:48:06 dist-test-slave-5bvn krb5kdc[15386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.130@KRBTEST.COM
I20260620 21:48:06.226353 16968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:06.208697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:45423 (local address 127.1.155.129:57895)
0620 21:48:06.209003 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:06.209015 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:06.209072 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:06.211088 (+ 2016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:06.211095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:06.211778 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:06.212690 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:06.212712 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:06.214761 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:06.214764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:06.216466 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:06.216478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:06.217524 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:06.217528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:06.217800 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:06.218629 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:06.218642 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:06.220604 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:06.223186 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:06.223192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:06.223206 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:06.223657 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:06.224116 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:06.224120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:06.224122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:06.224357 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:06.225138 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:06.225144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:06.225516 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 21:48:06.225746 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 21:48:06.226109 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":519,"spinlock_wait_cycles":25856,"thread_start_us":61,"threads_started":1}
I20260620 21:48:06.226393 16971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:06.208901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:57895 (local address 127.1.155.131:45423)
0620 21:48:06.209232 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:06.209251 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:06.209280 (+ 29us) server_negotiation.cc:407] Connection header received
0620 21:48:06.210185 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:06.210193 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:06.210630 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:06.210962 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:06.212916 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:06.214537 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:06.216604 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:06.217371 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:06.217548 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:06.220867 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:06.220872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:06.220891 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:06.220919 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:06.223043 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:06.223783 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:06.223789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:06.223795 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:06.223971 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:06.224469 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:06.224473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:06.224475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:06.224985 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:06.225118 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:06.225917 (+ 799us) server_negotiation.cc:299] Negotiation successful
0620 21:48:06.226208 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":74,"threads_started":1}
I20260620 21:48:06.226394 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:06.208780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:60871 (local address 127.1.155.130:34019)
0620 21:48:06.209216 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:06.209232 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:06.209278 (+ 46us) server_negotiation.cc:407] Connection header received
0620 21:48:06.210185 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:06.210193 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:06.210630 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:06.210965 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:06.212916 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:06.214533 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:06.216602 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:06.217371 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:06.217539 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:06.221126 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:06.221136 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:06.221150 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:06.221187 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:06.223043 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:06.223783 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:06.223789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:06.223795 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:06.223971 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:06.224469 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:06.224473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:06.224474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:06.224984 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:06.225118 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:06.225917 (+ 799us) server_negotiation.cc:299] Negotiation successful
0620 21:48:06.226208 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":81,"threads_started":1}
I20260620 21:48:06.226354 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:06.208601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:34019 (local address 127.1.155.129:60871)
0620 21:48:06.208918 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:06.208934 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:06.209057 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:06.211088 (+ 2031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:06.211095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:06.211778 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:06.212690 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:06.212712 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:06.214760 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:06.214764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:06.216464 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:06.216478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:06.217524 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:06.217528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:06.217805 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:06.218603 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:06.218634 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:06.220931 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:06.223186 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:06.223192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:06.223206 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:06.223657 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:06.224116 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:06.224120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:06.224122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:06.224357 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:06.225137 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:06.225144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:06.225516 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 21:48:06.225746 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 21:48:06.226161 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":530,"spinlock_wait_cycles":48896,"thread_start_us":71,"threads_started":1}
W20260620 21:48:06.229169 16580 leader_election.cc:341] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423): Not found: Tablet not found: 2b378879a5b84352b926970d341d26b1
W20260620 21:48:06.229519 16582 leader_election.cc:341] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44286227758740a0944e5091566231b8 (127.1.155.130:34019): Not found: Tablet not found: 2b378879a5b84352b926970d341d26b1
I20260620 21:48:06.229609 16582 leader_election.cc:302] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89d4f97e5c28415d9633552e764362a5; no voters: 44286227758740a0944e5091566231b8, e4347d5750a24d21a19dbd71663b08bc
I20260620 21:48:06.229827 16967 raft_consensus.cc:2726] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:48:06.397980 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:06.397632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:40415)
0620 21:48:06.397774 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:06.397888 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 21:48:06.560084 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:06.559568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:37559)
0620 21:48:06.559872 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:06.559947 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260620 21:48:06.768813 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:06.768414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:36847)
0620 21:48:06.768671 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:06.768725 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
I20260620 21:48:07.399113 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:07.398717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:41935)
0620 21:48:07.398965 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:07.399021 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260620 21:48:07.561404 16975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:07.560927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:55873)
0620 21:48:07.561247 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:07.561310 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":74,"threads_started":1}
I20260620 21:48:07.770148 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:07.769672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:40881)
0620 21:48:07.769948 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:07.770026 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":93,"threads_started":1}
I20260620 21:48:08.151409 16977 raft_consensus.cc:482] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:48:08.151577 16977 raft_consensus.cc:504] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:08.152017 16977 leader_election.cc:288] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44286227758740a0944e5091566231b8 (127.1.155.130:34019), e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423)
W20260620 21:48:08.152935 16582 leader_election.cc:341] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44286227758740a0944e5091566231b8 (127.1.155.130:34019): Not found: Tablet not found: 2b378879a5b84352b926970d341d26b1
W20260620 21:48:08.153038 16580 leader_election.cc:341] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423): Not found: Tablet not found: 2b378879a5b84352b926970d341d26b1
I20260620 21:48:08.153121 16580 leader_election.cc:302] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89d4f97e5c28415d9633552e764362a5; no voters: 44286227758740a0944e5091566231b8, e4347d5750a24d21a19dbd71663b08bc
I20260620 21:48:08.153260 16977 raft_consensus.cc:2726] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:48:08.400424 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:08.399969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:55983)
0620 21:48:08.400246 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:08.400322 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":95,"threads_started":1}
I20260620 21:48:08.562860 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:08.562327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:45519)
0620 21:48:08.562669 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:08.562744 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
I20260620 21:48:08.771306 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:08.770893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:52217)
0620 21:48:08.771158 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:08.771217 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":93,"threads_started":1}
I20260620 21:48:09.401917 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:09.401377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:36255)
0620 21:48:09.401704 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:09.401765 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":89,"threads_started":1}
I20260620 21:48:09.564926 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:09.564402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:57695)
0620 21:48:09.564707 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:09.564807 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":89,"threads_started":1}
I20260620 21:48:09.772552 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:09.772134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:53313)
0620 21:48:09.772397 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:09.772459 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
I20260620 21:48:10.403126 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:10.402690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:53573)
0620 21:48:10.402970 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:10.403033 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260620 21:48:10.566190 16985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:10.565703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:55951)
0620 21:48:10.566033 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:10.566092 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":71,"threads_started":1}
I20260620 21:48:10.738214 16986 raft_consensus.cc:482] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:48:10.738344 16986 raft_consensus.cc:504] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:10.738827 16986 leader_election.cc:288] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44286227758740a0944e5091566231b8 (127.1.155.130:34019), e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423)
W20260620 21:48:10.739678 16582 leader_election.cc:341] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44286227758740a0944e5091566231b8 (127.1.155.130:34019): Not found: Tablet not found: 2b378879a5b84352b926970d341d26b1
W20260620 21:48:10.739773 16580 leader_election.cc:341] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423): Not found: Tablet not found: 2b378879a5b84352b926970d341d26b1
I20260620 21:48:10.739879 16580 leader_election.cc:302] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89d4f97e5c28415d9633552e764362a5; no voters: 44286227758740a0944e5091566231b8, e4347d5750a24d21a19dbd71663b08bc
I20260620 21:48:10.740046 16986 raft_consensus.cc:2726] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:48:10.773777 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:10.773338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:50999)
0620 21:48:10.773618 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:10.773687 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":89,"threads_started":1}
I20260620 21:48:11.404404 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:11.403950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:35545)
0620 21:48:11.404239 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:11.404309 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":85,"threads_started":1}
I20260620 21:48:11.567314 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:11.566875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:36585)
0620 21:48:11.567164 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:11.567220 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":71,"threads_started":1}
I20260620 21:48:11.775023 16990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:11.774557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:39415)
0620 21:48:11.774879 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:11.774935 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260620 21:48:12.405920 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:12.405391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:33215)
0620 21:48:12.405692 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:12.405749 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":100,"threads_started":1}
I20260620 21:48:12.568527 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:12.568111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:42081)
0620 21:48:12.568371 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:12.568429 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20260620 21:48:12.776330 16993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:12.775889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:41139)
0620 21:48:12.776162 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:12.776224 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":80,"threads_started":1}
I20260620 21:48:13.407315 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:13.406826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:49551)
0620 21:48:13.407137 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:13.407205 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":95,"threads_started":1}
I20260620 21:48:13.569862 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:13.569351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:44763)
0620 21:48:13.569648 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:13.569713 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":85,"threads_started":1}
I20260620 21:48:13.777940 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:13.777315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:46653)
0620 21:48:13.777692 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:13.777757 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":110,"threads_started":1}
I20260620 21:48:13.782111 16997 raft_consensus.cc:482] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:48:13.782243 16997 raft_consensus.cc:504] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:13.782634 16997 leader_election.cc:288] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44286227758740a0944e5091566231b8 (127.1.155.130:34019), e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423)
W20260620 21:48:13.783536 16580 leader_election.cc:341] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423): Not found: Tablet not found: 2b378879a5b84352b926970d341d26b1
W20260620 21:48:13.783650 16582 leader_election.cc:341] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44286227758740a0944e5091566231b8 (127.1.155.130:34019): Not found: Tablet not found: 2b378879a5b84352b926970d341d26b1
I20260620 21:48:13.783752 16582 leader_election.cc:302] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89d4f97e5c28415d9633552e764362a5; no voters: 44286227758740a0944e5091566231b8, e4347d5750a24d21a19dbd71663b08bc
I20260620 21:48:13.783959 16997 raft_consensus.cc:2726] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:48:14.409036 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:14.408423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:47841)
0620 21:48:14.408769 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:14.408835 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":106,"threads_started":1}
I20260620 21:48:14.571110 16999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:14.570707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:49061)
0620 21:48:14.570956 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:14.571015 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20260620 21:48:14.779254 17000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:14.778813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:52209)
0620 21:48:14.779086 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:14.779162 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":78,"threads_started":1}
I20260620 21:48:15.410526 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:15.410096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:34181)
0620 21:48:15.410372 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:15.410430 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260620 21:48:15.572645 17002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:15.572145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:35975)
0620 21:48:15.572401 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:15.572505 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
I20260620 21:48:15.780691 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:15.780219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:52545)
0620 21:48:15.780532 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:15.780593 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":113,"threads_started":1}
I20260620 21:48:16.411784 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:16.411337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:58363)
0620 21:48:16.411634 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:16.411693 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":107,"threads_started":1}
I20260620 21:48:16.573992 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:16.573540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:38611)
0620 21:48:16.573779 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:16.573886 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
I20260620 21:48:16.781971 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:16.781503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:49355)
0620 21:48:16.781763 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:16.781820 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
I20260620 21:48:17.368996 17007 raft_consensus.cc:482] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:48:17.369124 17007 raft_consensus.cc:504] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44286227758740a0944e5091566231b8" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 34019 } } peers { permanent_uuid: "e4347d5750a24d21a19dbd71663b08bc" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45423 } } peers { permanent_uuid: "89d4f97e5c28415d9633552e764362a5" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34437 } }
I20260620 21:48:17.369606 17007 leader_election.cc:288] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44286227758740a0944e5091566231b8 (127.1.155.130:34019), e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423)
W20260620 21:48:17.370374 16582 leader_election.cc:341] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44286227758740a0944e5091566231b8 (127.1.155.130:34019): Not found: Tablet not found: 2b378879a5b84352b926970d341d26b1
W20260620 21:48:17.370478 16580 leader_election.cc:341] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4347d5750a24d21a19dbd71663b08bc (127.1.155.131:45423): Not found: Tablet not found: 2b378879a5b84352b926970d341d26b1
I20260620 21:48:17.370676 16582 leader_election.cc:302] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89d4f97e5c28415d9633552e764362a5; no voters: 44286227758740a0944e5091566231b8, e4347d5750a24d21a19dbd71663b08bc
I20260620 21:48:17.370817 17007 raft_consensus.cc:2726] T 2b378879a5b84352b926970d341d26b1 P 89d4f97e5c28415d9633552e764362a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:48:17.412964 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:17.412535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:35937)
0620 21:48:17.412812 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:17.412869 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
I20260620 21:48:17.575390 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:17.574940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:41529)
0620 21:48:17.575190 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:17.575269 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
I20260620 21:48:17.783181 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:17.782757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:54311)
0620 21:48:17.783036 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:17.783089 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
I20260620 21:48:18.414281 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:18.413863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:40017)
0620 21:48:18.414121 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:18.414184 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
I20260620 21:48:18.576512 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:18.576105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:52939)
0620 21:48:18.576366 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:18.576418 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20260620 21:48:18.784492 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:18.783929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:56537)
0620 21:48:18.784290 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:18.784364 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":79,"threads_started":1}
I20260620 21:48:19.415658 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:19.415237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.129:34347)
0620 21:48:19.415504 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:19.415563 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
I20260620 21:48:19.578009 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:19.577387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.130:57479)
0620 21:48:19.577689 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:19.577749 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":72,"threads_started":1}
I20260620 21:48:19.777005 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 16530
I20260620 21:48:19.782956 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 16697
I20260620 21:48:19.785976 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:19.785469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39637 (local address 127.1.155.131:42549)
0620 21:48:19.785742 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:19.785819 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260620 21:48:19.788610 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 16831
2026-06-20T21:48:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19498 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:48:19 dist-test-slave-5bvn krb5kdc[17019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:48:19 dist-test-slave-5bvn krb5kdc[17019](info): set up 2 sockets
Jun 20 21:48:19 dist-test-slave-5bvn krb5kdc[17019](info): commencing operation
krb5kdc: starting...
W20260620 21:48:20.858148 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:48:20 dist-test-slave-5bvn krb5kdc[17019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:48:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:48:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:48:20.980928 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:36535
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:36535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:48:21.091936 17032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:21.092212 17032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:21.092254 17032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:21.096216 17032 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 21:48:21.096272 17032 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 21:48:21.096287 17032 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 21:48:21.096302 17032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:48:21.096320 17032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:21.096338 17032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:21.096354 17032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:48:21.096369 17032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:48:21.101786 17032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:36535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:36535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:21.102885 17032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:21.103906 17032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:21.112701 17040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:21.112815 17037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:21.112701 17038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:21.113004 17032 server_base.cc:1034] running on GCE node
I20260620 21:48:21.113678 17032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:21.114991 17032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:21.116171 17032 hybrid_clock.cc:648] HybridClock initialized: now 1781992101116116 us; error 67 us; skew 500 ppm
Jun 20 21:48:21 dist-test-slave-5bvn krb5kdc[17019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992101, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:21.119334 17032 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:48:21.120271 17032 webserver.cc:460] Webserver started at http://127.1.155.190:34045/ using document root <none> and password file <none>
I20260620 21:48:21.120831 17032 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:21.120899 17032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:21.121076 17032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:48:21.123342 17032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "503b4dc73f344f549d96e55154f64c5c"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "83dbbccbd54a318b823008d995c84636"
server_key_iv: "55f2e34121d966247362c5f5bd0bcce3"
server_key_version: "encryptionkey@0"
I20260620 21:48:21.123924 17032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "503b4dc73f344f549d96e55154f64c5c"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "83dbbccbd54a318b823008d995c84636"
server_key_iv: "55f2e34121d966247362c5f5bd0bcce3"
server_key_version: "encryptionkey@0"
I20260620 21:48:21.128018 17032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 21:48:21.130712 17047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:21.131958 17032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:48:21.132110 17032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "503b4dc73f344f549d96e55154f64c5c"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "83dbbccbd54a318b823008d995c84636"
server_key_iv: "55f2e34121d966247362c5f5bd0bcce3"
server_key_version: "encryptionkey@0"
I20260620 21:48:21.132226 17032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:21.154214 17032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:21.157167 17032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:21.157364 17032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:21.164582 17032 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:36535
I20260620 21:48:21.164867 17099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:36535 every 8 connection(s)
I20260620 21:48:21.165637 17032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:48:21.167922 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 17032
I20260620 21:48:21.168054 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:48:21.168361 1646 external_mini_cluster.cc:1428] Setting key a9f196e1ff601ba1a81a22f3bfe26c1c
I20260620 21:48:21.168880 17100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:48:21.176617 17100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Bootstrap starting.
Jun 20 21:48:21 dist-test-slave-5bvn krb5kdc[17019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:48:21.178952 17100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Neither blocks nor log segments found. Creating new log.
I20260620 21:48:21.179701 17100 log.cc:824] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:21.181798 17100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: No bootstrap required, opened a new log
I20260620 21:48:21.183786 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:21.169919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.1.155.190:36535)
0620 21:48:21.170436 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:21.170457 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:21.170499 (+ 42us) server_negotiation.cc:407] Connection header received
0620 21:48:21.171232 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:21.171249 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:21.171594 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:21.171894 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:21.172884 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.174459 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.175668 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.176163 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.176378 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:21.178982 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:21.178987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:21.179001 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:21.179035 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:21.181003 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:21.181452 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:21.181458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:21.181464 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:21.181581 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:21.181878 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:21.181882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:21.181884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:21.182313 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:21.182511 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:21.182806 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 21:48:21.183108 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":153,"threads_started":1}
I20260620 21:48:21.185768 17100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } }
I20260620 21:48:21.186108 17100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:48:21.186208 17100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 503b4dc73f344f549d96e55154f64c5c, State: Initialized, Role: FOLLOWER
I20260620 21:48:21.186686 17100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } }
I20260620 21:48:21.186847 17100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:48:21.186934 17100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:48:21.187038 17100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:48:21.188170 17100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } }
I20260620 21:48:21.188531 17100 leader_election.cc:302] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 503b4dc73f344f549d96e55154f64c5c; no voters:
I20260620 21:48:21.188858 17100 leader_election.cc:288] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:48:21.189050 17105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:48:21.189390 17105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 1 LEADER]: Becoming Leader. State: Replica: 503b4dc73f344f549d96e55154f64c5c, State: Running, Role: LEADER
I20260620 21:48:21.189728 17105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } }
I20260620 21:48:21.190162 17100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:48:21.191475 17106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "503b4dc73f344f549d96e55154f64c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } } }
I20260620 21:48:21.191488 17107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 503b4dc73f344f549d96e55154f64c5c. Latest consensus state: current_term: 1 leader_uuid: "503b4dc73f344f549d96e55154f64c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } } }
I20260620 21:48:21.191615 17107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [sys.catalog]: This master's current role is: LEADER
I20260620 21:48:21.191615 17106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [sys.catalog]: This master's current role is: LEADER
I20260620 21:48:21.192052 17116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:48:21.195530 17116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:48:21.201160 17116 catalog_manager.cc:1347] Generated new cluster ID: 9bdd1bb87aef45b1aef39ff06c61c214
I20260620 21:48:21.201231 17116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:48:21.217146 17116 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:48:21.218210 17116 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:48:21.227414 17116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Generated new TSK 0
I20260620 21:48:21.228135 17116 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:48:21.262671 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:36535
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:48:21.390272 17126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:21.390544 17126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:21.390587 17126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:21.394752 17126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:21.394805 17126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:21.394863 17126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:48:21.400111 17126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:21.401099 17126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:21.402076 17126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:21.412111 17134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:21.412115 17132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:21.412111 17131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:21.412371 17126 server_base.cc:1034] running on GCE node
I20260620 21:48:21.412910 17126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:21.413502 17126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:21.414678 17126 hybrid_clock.cc:648] HybridClock initialized: now 1781992101414663 us; error 26 us; skew 500 ppm
Jun 20 21:48:21 dist-test-slave-5bvn krb5kdc[17019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992101, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:21.417377 17126 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:48:21.418311 17126 webserver.cc:460] Webserver started at http://127.1.155.129:34557/ using document root <none> and password file <none>
I20260620 21:48:21.418823 17126 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:21.418895 17126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:21.419109 17126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:48:21.421298 17126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "3ab36752164b41e282202c4762838338"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "944670c29946f49eaac942bb5d01c5c5"
server_key_iv: "47b81451ac28f1bd6eec8d31543619f7"
server_key_version: "encryptionkey@0"
I20260620 21:48:21.421968 17126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "3ab36752164b41e282202c4762838338"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "944670c29946f49eaac942bb5d01c5c5"
server_key_iv: "47b81451ac28f1bd6eec8d31543619f7"
server_key_version: "encryptionkey@0"
I20260620 21:48:21.425606 17126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:48:21.428052 17141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:21.429279 17126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:48:21.429414 17126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "3ab36752164b41e282202c4762838338"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "944670c29946f49eaac942bb5d01c5c5"
server_key_iv: "47b81451ac28f1bd6eec8d31543619f7"
server_key_version: "encryptionkey@0"
I20260620 21:48:21.429525 17126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:21.448083 17126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:21.451576 17126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:21.451813 17126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:21.452435 17126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:48:21.453270 17126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:48:21.453344 17126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:21.453413 17126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:48:21.453464 17126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:21.463312 17126 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:36685
I20260620 21:48:21.463348 17254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:36685 every 8 connection(s)
I20260620 21:48:21.464207 17126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:48:21.469946 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 17126
I20260620 21:48:21.470104 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:48:21.470407 1646 external_mini_cluster.cc:1428] Setting key be6c5ae8b36cdeb480e36891772befef
Jun 20 21:48:21 dist-test-slave-5bvn krb5kdc[17019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992101, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:48:21.479756 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:21.466141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:48007 (local address 127.1.155.190:36535)
0620 21:48:21.466286 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:21.466290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:21.467147 (+ 857us) server_negotiation.cc:407] Connection header received
0620 21:48:21.468435 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:21.468438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:21.468496 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:21.468613 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:21.469732 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.470448 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.472072 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.472400 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.472483 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:21.475682 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:21.475685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:21.475687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:21.475710 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:21.477286 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:21.477970 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:21.477972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:21.477974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:21.478049 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:21.478456 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:21.478459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:21.478460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:21.478655 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:21.478729 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:21.479429 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 21:48:21.479543 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:48:21.480841 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:21.466544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:48007)
0620 21:48:21.466988 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:21.467036 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:21.468217 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:21.468778 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:21.468784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:21.469131 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:21.469588 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:21.469600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.470596 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.470600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:21.471911 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:21.471923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.472563 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.472570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:21.472797 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:21.473423 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:21.473449 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:21.475486 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:21.477406 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:21.477415 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:21.477436 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:21.477860 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:21.478149 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:21.478154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:21.478156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:21.478357 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:21.478764 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:21.478773 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:21.479120 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 21:48:21.479353 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 21:48:21.479722 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
I20260620 21:48:21.482256 17255 heartbeater.cc:347] Connected to a master server at 127.1.155.190:36535
I20260620 21:48:21.482477 17255 heartbeater.cc:464] Registering TS with master...
I20260620 21:48:21.482940 17255 heartbeater.cc:511] Master 127.1.155.190:36535 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
I20260620 21:48:21.484577 17064 ts_manager.cc:194] Registered new tserver with Master: 3ab36752164b41e282202c4762838338 (127.1.155.129:36685)
I20260620 21:48:21.485944 17064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:48007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:48:21.501502 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:36535
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:48:21.612535 17260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:21.612792 17260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:21.612835 17260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:21.616890 17260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:21.616957 17260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:21.617018 17260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:48:21.622361 17260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:21.623373 17260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:21.624305 17260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:21.633898 17266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:21.633899 17265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:21.634270 17260 server_base.cc:1034] running on GCE node
W20260620 21:48:21.633898 17268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:21.634794 17260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:21.635445 17260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:21.636641 17260 hybrid_clock.cc:648] HybridClock initialized: now 1781992101636636 us; error 45 us; skew 500 ppm
Jun 20 21:48:21 dist-test-slave-5bvn krb5kdc[17019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992101, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:21.639479 17260 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:48:21.640379 17260 webserver.cc:460] Webserver started at http://127.1.155.130:40841/ using document root <none> and password file <none>
I20260620 21:48:21.640938 17260 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:21.641026 17260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:21.641258 17260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:48:21.643313 17260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "49f3320599cc4370b7ec5e84a692450d"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "412f77cc7d84bdde00bcd588a3b7483c"
server_key_iv: "624c008104d6013887d3d1fc875d5aa3"
server_key_version: "encryptionkey@0"
I20260620 21:48:21.643868 17260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "49f3320599cc4370b7ec5e84a692450d"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "412f77cc7d84bdde00bcd588a3b7483c"
server_key_iv: "624c008104d6013887d3d1fc875d5aa3"
server_key_version: "encryptionkey@0"
I20260620 21:48:21.647574 17260 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:48:21.650106 17275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:21.651345 17260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:48:21.651489 17260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "49f3320599cc4370b7ec5e84a692450d"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "412f77cc7d84bdde00bcd588a3b7483c"
server_key_iv: "624c008104d6013887d3d1fc875d5aa3"
server_key_version: "encryptionkey@0"
I20260620 21:48:21.651608 17260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:21.665130 17260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:21.668474 17260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:21.668707 17260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:21.669312 17260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:48:21.670188 17260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:48:21.670269 17260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:21.670343 17260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:48:21.670397 17260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:21.679208 17260 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:43845
I20260620 21:48:21.679239 17388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:43845 every 8 connection(s)
I20260620 21:48:21.680068 17260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:48:21.688053 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 17260
I20260620 21:48:21.688198 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:48:21.688521 1646 external_mini_cluster.cc:1428] Setting key 6b055de657ae97f42a96ffa2899d6216
Jun 20 21:48:21 dist-test-slave-5bvn krb5kdc[17019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992101, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:48:21.694326 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:21.681951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:33549 (local address 127.1.155.190:36535)
0620 21:48:21.682084 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:21.682087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:21.682840 (+ 753us) server_negotiation.cc:407] Connection header received
0620 21:48:21.684291 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:21.684295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:21.684354 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:21.684458 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:21.685589 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.686176 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.687290 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.687509 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.687619 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:21.690360 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:21.690363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:21.690365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:21.690387 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:21.692235 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:21.692794 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:21.692799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:21.692801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:21.692888 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:21.693219 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:21.693224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:21.693228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:21.693412 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:21.693517 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:21.694051 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 21:48:21.694170 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:48:21.695189 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:21.682263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:33549)
0620 21:48:21.682678 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:21.682735 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:21.684078 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:21.684623 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:21.684629 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:21.684985 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:21.685447 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:21.685459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.686314 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.686318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:21.687161 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:21.687170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.687654 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.687658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:21.687859 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:21.688525 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:21.688548 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:21.690180 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:21.692348 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:21.692354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:21.692368 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:21.692691 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:21.692984 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:21.692988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:21.692990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:21.693126 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:21.693516 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:21.693522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:21.693768 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 21:48:21.694009 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 21:48:21.694336 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":100,"threads_started":1}
I20260620 21:48:21.696524 17389 heartbeater.cc:347] Connected to a master server at 127.1.155.190:36535
I20260620 21:48:21.696818 17389 heartbeater.cc:464] Registering TS with master...
I20260620 21:48:21.697355 17389 heartbeater.cc:511] Master 127.1.155.190:36535 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
I20260620 21:48:21.698515 17064 ts_manager.cc:194] Registered new tserver with Master: 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845)
I20260620 21:48:21.699134 17064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:33549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:48:21.716583 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:36535
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:48:21.830269 17394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:21.830506 17394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:21.830551 17394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:21.834815 17394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:21.834879 17394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:21.834939 17394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:48:21.840174 17394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:21.841109 17394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:21.842152 17394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:21.851512 17400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:21.851533 17399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:21.851511 17402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:21.851689 17394 server_base.cc:1034] running on GCE node
I20260620 21:48:21.852264 17394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:21.852875 17394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:21.854029 17394 hybrid_clock.cc:648] HybridClock initialized: now 1781992101853999 us; error 41 us; skew 500 ppm
Jun 20 21:48:21 dist-test-slave-5bvn krb5kdc[17019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992101, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:21.856740 17394 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:48:21.857560 17394 webserver.cc:460] Webserver started at http://127.1.155.131:33817/ using document root <none> and password file <none>
I20260620 21:48:21.858163 17394 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:21.858253 17394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:21.858485 17394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:48:21.860430 17394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "f29d3ae3dfc14c5282ecfc045bd73991"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "9d674599504f5abd7c56b9d4c8335a32"
server_key_iv: "6409b999d03d7b27b03c80cafaa3c25f"
server_key_version: "encryptionkey@0"
I20260620 21:48:21.860981 17394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "f29d3ae3dfc14c5282ecfc045bd73991"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "9d674599504f5abd7c56b9d4c8335a32"
server_key_iv: "6409b999d03d7b27b03c80cafaa3c25f"
server_key_version: "encryptionkey@0"
I20260620 21:48:21.864660 17394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 21:48:21.867081 17409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:21.868273 17394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:48:21.868419 17394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "f29d3ae3dfc14c5282ecfc045bd73991"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "9d674599504f5abd7c56b9d4c8335a32"
server_key_iv: "6409b999d03d7b27b03c80cafaa3c25f"
server_key_version: "encryptionkey@0"
I20260620 21:48:21.868537 17394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:21.893405 17394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:21.896363 17394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:21.896570 17394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:21.897156 17394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:48:21.898079 17394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:48:21.898151 17394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:21.898249 17394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:48:21.898293 17394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:21.907310 17394 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:43431
I20260620 21:48:21.907330 17522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:43431 every 8 connection(s)
I20260620 21:48:21.908193 17394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:48:21.913378 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 17394
I20260620 21:48:21.913514 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:48:21.913815 1646 external_mini_cluster.cc:1428] Setting key b74d6fb37a657097567c93fee2197018
Jun 20 21:48:21 dist-test-slave-5bvn krb5kdc[17019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992101, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:48:21.922919 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:21.910220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:44835 (local address 127.1.155.190:36535)
0620 21:48:21.910366 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:21.910371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:21.911316 (+ 945us) server_negotiation.cc:407] Connection header received
0620 21:48:21.912629 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:21.912632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:21.912678 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:21.912759 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:21.914320 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.914960 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.916411 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.916639 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.916728 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:21.919459 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:21.919462 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:21.919464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:21.919487 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:21.920885 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:21.921396 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:21.921398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:21.921400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:21.921474 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:21.921814 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:21.921816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:21.921818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:21.922079 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:21.922166 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:21.922667 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 21:48:21.922765 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:48:21.923727 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:21.910683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:44835)
0620 21:48:21.911153 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:21.911195 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:21.912424 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:21.912922 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:21.912929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:21.913434 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:21.914150 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:21.914180 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.915103 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.915106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:21.916285 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:21.916294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.916788 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.916791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:21.917131 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:21.917744 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:21.917761 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:21.919298 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:21.921017 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:21.921022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:21.921035 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:21.921297 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:21.921555 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:21.921557 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:21.921558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:21.921688 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:21.922187 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:21.922196 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:21.922456 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 21:48:21.922620 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 21:48:21.922872 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":96,"threads_started":1}
I20260620 21:48:21.924888 17523 heartbeater.cc:347] Connected to a master server at 127.1.155.190:36535
I20260620 21:48:21.925153 17523 heartbeater.cc:464] Registering TS with master...
I20260620 21:48:21.925640 17523 heartbeater.cc:511] Master 127.1.155.190:36535 requested a full tablet report, sending...
I20260620 21:48:21.926916 17064 ts_manager.cc:194] Registered new tserver with Master: f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431)
I20260620 21:48:21.927611 17064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:44835
I20260620 21:48:21.928062 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:48:21.937032 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:21.929988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.1.155.190:36535)
0620 21:48:21.930116 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:21.930121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:21.930296 (+ 175us) server_negotiation.cc:407] Connection header received
0620 21:48:21.930379 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:21.930381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:21.930417 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:21.930480 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:21.931208 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.931687 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.932477 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.932674 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.932749 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:21.934096 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:21.934099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:21.934101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:21.934123 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:21.935361 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:21.935725 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:21.935728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:21.935729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:21.935833 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:21.936102 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:21.936105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:21.936106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:21.936365 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:21.936488 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:21.936688 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 21:48:21.936779 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:48:21.939909 17064 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:48:21.942834 17064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:48:21.958074 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:21.951613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:36685 (local address 127.0.0.1:35760)
0620 21:48:21.952555 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:21.952590 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:21.952725 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:21.953476 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:21.953482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:21.953504 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:21.953642 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:21.953649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.955352 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.955355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:21.956714 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:21.956723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.957515 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.957519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:21.957621 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:21.957656 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 21:48:21.957764 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 21:48:21.957910 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":155,"spinlock_wait_cycles":11392,"thread_start_us":88,"threads_started":1}
I20260620 21:48:21.958076 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:21.952196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.1.155.129:36685)
0620 21:48:21.952966 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:21.952985 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:21.953009 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:48:21.953106 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:21.953113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:21.953309 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:21.953468 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:21.953774 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.955230 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.956812 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.957378 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.957515 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:21.957664 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:21.957791 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 21:48:21.957906 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":70,"threads_started":1}
I20260620 21:48:21.958452 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:21.951871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:43845 (local address 127.0.0.1:42894)
0620 21:48:21.952965 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:21.952975 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:21.953073 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:21.953644 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:21.953647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:21.953675 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:21.953820 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:21.953825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.954925 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.954931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:21.956617 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:21.956627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.957947 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.957950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:21.958090 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:21.958120 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 21:48:21.958216 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 21:48:21.958311 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":108,"spinlock_wait_cycles":8960,"thread_start_us":100,"threads_started":1}
I20260620 21:48:21.958869 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:21.951540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:43431 (local address 127.0.0.1:53912)
0620 21:48:21.952539 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:21.952590 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:21.952725 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:21.953644 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:21.953647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:21.953672 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:21.953817 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:21.953823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.955283 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.955289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:21.956473 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:21.956482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.957924 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.957930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:21.958590 (+ 660us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:21.958614 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:48:21.958682 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 21:48:21.958744 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":557,"spinlock_wait_cycles":6400,"thread_start_us":89,"threads_started":1}
I20260620 21:48:21.958892 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:21.952302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42894 (local address 127.1.155.130:43845)
0620 21:48:21.952700 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:21.952712 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:21.953063 (+ 351us) server_negotiation.cc:407] Connection header received
0620 21:48:21.953182 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:21.953187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:21.953338 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:21.953497 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:21.954000 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.954791 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.956744 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.957635 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.957724 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:21.957897 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:21.958601 (+ 704us) server_negotiation.cc:299] Negotiation successful
0620 21:48:21.958705 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":106,"threads_started":1}
I20260620 21:48:21.959336 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:21.951665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53912 (local address 127.1.155.131:43431)
0620 21:48:21.953056 (+ 1391us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:21.953067 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:21.953086 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:48:21.953148 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:21.953153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:21.953309 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:21.953468 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:21.953970 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.955122 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.956593 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:21.957579 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:21.957708 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:21.958022 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:21.959098 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0620 21:48:21.959203 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1296,"thread_start_us":148,"threads_started":1}
I20260620 21:48:21.961176 17189 tablet_service.cc:1467] Processing CreateTablet for tablet cec0d938cd77492e8e136cd0848a97a2 (DEFAULT_TABLE table=test-table [id=116eaa23a58443adb20a12d556cfdb52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:48:21.961175 17323 tablet_service.cc:1467] Processing CreateTablet for tablet cec0d938cd77492e8e136cd0848a97a2 (DEFAULT_TABLE table=test-table [id=116eaa23a58443adb20a12d556cfdb52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:48:21.962141 17323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec0d938cd77492e8e136cd0848a97a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:48:21.962167 17189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec0d938cd77492e8e136cd0848a97a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:48:21.963534 17457 tablet_service.cc:1467] Processing CreateTablet for tablet cec0d938cd77492e8e136cd0848a97a2 (DEFAULT_TABLE table=test-table [id=116eaa23a58443adb20a12d556cfdb52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:48:21.964511 17457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec0d938cd77492e8e136cd0848a97a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:48:21.968323 17540 tablet_bootstrap.cc:492] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d: Bootstrap starting.
I20260620 21:48:21.970031 17541 tablet_bootstrap.cc:492] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: Bootstrap starting.
I20260620 21:48:21.971107 17540 tablet_bootstrap.cc:654] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d: Neither blocks nor log segments found. Creating new log.
I20260620 21:48:21.971148 17542 tablet_bootstrap.cc:492] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991: Bootstrap starting.
I20260620 21:48:21.972014 17540 log.cc:824] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:21.972506 17541 tablet_bootstrap.cc:654] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: Neither blocks nor log segments found. Creating new log.
I20260620 21:48:21.973637 17542 tablet_bootstrap.cc:654] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991: Neither blocks nor log segments found. Creating new log.
I20260620 21:48:21.973676 17541 log.cc:824] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:21.974300 17540 tablet_bootstrap.cc:492] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d: No bootstrap required, opened a new log
I20260620 21:48:21.974509 17540 ts_tablet_manager.cc:1397] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:48:21.974525 17542 log.cc:824] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:21.975615 17541 tablet_bootstrap.cc:492] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: No bootstrap required, opened a new log
I20260620 21:48:21.975823 17541 ts_tablet_manager.cc:1397] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:48:21.976454 17542 tablet_bootstrap.cc:492] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991: No bootstrap required, opened a new log
I20260620 21:48:21.976629 17542 ts_tablet_manager.cc:1397] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 21:48:21.978403 17540 raft_consensus.cc:348] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } } peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } }
I20260620 21:48:21.978703 17540 raft_consensus.cc:374] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:48:21.978793 17540 raft_consensus.cc:729] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f3320599cc4370b7ec5e84a692450d, State: Initialized, Role: FOLLOWER
I20260620 21:48:21.978912 17541 raft_consensus.cc:348] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } } peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } }
I20260620 21:48:21.979166 17541 raft_consensus.cc:374] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:48:21.979250 17541 raft_consensus.cc:729] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab36752164b41e282202c4762838338, State: Initialized, Role: FOLLOWER
I20260620 21:48:21.979280 17540 consensus_queue.cc:260] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } } peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } }
I20260620 21:48:21.979700 17541 consensus_queue.cc:260] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } } peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } }
I20260620 21:48:21.979964 17389 heartbeater.cc:503] Master 127.1.155.190:36535 was elected leader, sending a full tablet report...
I20260620 21:48:21.980314 17540 ts_tablet_manager.cc:1428] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:48:21.980932 17541 ts_tablet_manager.cc:1428] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 21:48:21.980998 17255 heartbeater.cc:503] Master 127.1.155.190:36535 was elected leader, sending a full tablet report...
I20260620 21:48:21.982758 17542 raft_consensus.cc:348] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } } peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } }
I20260620 21:48:21.983054 17542 raft_consensus.cc:374] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:48:21.983152 17542 raft_consensus.cc:729] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29d3ae3dfc14c5282ecfc045bd73991, State: Initialized, Role: FOLLOWER
I20260620 21:48:21.983661 17542 consensus_queue.cc:260] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } } peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } }
I20260620 21:48:21.984356 17523 heartbeater.cc:503] Master 127.1.155.190:36535 was elected leader, sending a full tablet report...
I20260620 21:48:21.984774 17542 ts_tablet_manager.cc:1428] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260620 21:48:22.123399 17547 raft_consensus.cc:482] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:48:22.123687 17547 raft_consensus.cc:504] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } } peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } }
I20260620 21:48:22.124910 17547 leader_election.cc:288] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431), 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845)
I20260620 21:48:22.128975 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:22.125298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:40675 (local address 127.1.155.131:43431)
0620 21:48:22.125521 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:22.125527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:22.125629 (+ 102us) server_negotiation.cc:407] Connection header received
0620 21:48:22.125788 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:22.125792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:22.125862 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:22.125938 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:22.126228 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:22.126788 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:22.127934 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:22.128353 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:22.128496 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:22.128550 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:22.128774 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 21:48:22.128840 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260620 21:48:22.128999 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:22.125140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:43431 (local address 127.1.155.129:40675)
0620 21:48:22.125516 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:22.125532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:22.125661 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:22.125953 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:22.125956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:22.125976 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:22.126115 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:22.126123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:22.126915 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:22.126919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:22.127807 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:22.127816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:22.128481 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:22.128485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:22.128588 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:22.128631 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 21:48:22.128720 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 21:48:22.128841 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260620 21:48:22.129120 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:22.125415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:60555 (local address 127.1.155.130:43845)
0620 21:48:22.125630 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:22.125634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:22.125796 (+ 162us) server_negotiation.cc:407] Connection header received
0620 21:48:22.125965 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:22.125968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:22.126018 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:22.126102 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:22.126402 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:22.126968 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:22.128006 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:22.128471 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:22.128601 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:22.128657 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:22.128908 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 21:48:22.128979 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260620 21:48:22.129570 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:22.125289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:43845 (local address 127.1.155.129:60555)
0620 21:48:22.125672 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:22.125691 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:22.125812 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:22.126118 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:22.126120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:22.126135 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:22.126264 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:22.126270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:22.127084 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:22.127087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:22.127850 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:22.127858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:22.128595 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:22.128599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:22.128768 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:22.128793 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:48:22.128893 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 21:48:22.128965 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":25,"thread_start_us":105,"threads_started":1}
I20260620 21:48:22.129756 17477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec0d938cd77492e8e136cd0848a97a2" candidate_uuid: "3ab36752164b41e282202c4762838338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" is_pre_election: true
I20260620 21:48:22.130152 17477 raft_consensus.cc:2454] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ab36752164b41e282202c4762838338 in term 0.
I20260620 21:48:22.130237 17343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec0d938cd77492e8e136cd0848a97a2" candidate_uuid: "3ab36752164b41e282202c4762838338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49f3320599cc4370b7ec5e84a692450d" is_pre_election: true
I20260620 21:48:22.130532 17343 raft_consensus.cc:2454] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ab36752164b41e282202c4762838338 in term 0.
I20260620 21:48:22.130636 17142 leader_election.cc:302] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab36752164b41e282202c4762838338, f29d3ae3dfc14c5282ecfc045bd73991; no voters:
I20260620 21:48:22.130964 17547 raft_consensus.cc:2781] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:48:22.131060 17547 raft_consensus.cc:482] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:48:22.131102 17547 raft_consensus.cc:3037] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:48:22.132112 17547 raft_consensus.cc:504] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } } peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } }
I20260620 21:48:22.132500 17547 leader_election.cc:288] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 1 election: Requested vote from peers f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431), 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845)
I20260620 21:48:22.132833 17477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec0d938cd77492e8e136cd0848a97a2" candidate_uuid: "3ab36752164b41e282202c4762838338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29d3ae3dfc14c5282ecfc045bd73991"
I20260620 21:48:22.132866 17343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec0d938cd77492e8e136cd0848a97a2" candidate_uuid: "3ab36752164b41e282202c4762838338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49f3320599cc4370b7ec5e84a692450d"
I20260620 21:48:22.132942 17477 raft_consensus.cc:3037] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:48:22.133015 17343 raft_consensus.cc:3037] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:48:22.134199 17343 raft_consensus.cc:2454] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ab36752164b41e282202c4762838338 in term 1.
I20260620 21:48:22.134244 17477 raft_consensus.cc:2454] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ab36752164b41e282202c4762838338 in term 1.
I20260620 21:48:22.134500 17143 leader_election.cc:302] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab36752164b41e282202c4762838338, 49f3320599cc4370b7ec5e84a692450d; no voters:
I20260620 21:48:22.134693 17547 raft_consensus.cc:2781] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:48:22.134938 17547 raft_consensus.cc:686] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [term 1 LEADER]: Becoming Leader. State: Replica: 3ab36752164b41e282202c4762838338, State: Running, Role: LEADER
I20260620 21:48:22.135254 17547 consensus_queue.cc:237] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } } peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } }
I20260620 21:48:22.139233 17064 catalog_manager.cc:5495] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ab36752164b41e282202c4762838338 (127.1.155.129). New cstate: current_term: 1 leader_uuid: "3ab36752164b41e282202c4762838338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } health_report { overall_health: UNKNOWN } } }
W20260620 21:48:22.159174 17524 tablet.cc:2369] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:48:22.181319 17390 tablet.cc:2369] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:48:22.196312 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:22.192339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.1.155.129:36685)
0620 21:48:22.192517 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:22.192521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:22.192536 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:48:22.192626 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:22.192628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:22.192689 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:22.192787 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:48:22.193126 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:22.193667 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:22.194659 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:22.194892 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:22.194985 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:22.195380 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:48:22.195883 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:48:22.195988 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:22.196125 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 21:48:22.196187 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 21:48:22.204726 17343 raft_consensus.cc:1261] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d [term 1 FOLLOWER]: Refusing update from remote peer 3ab36752164b41e282202c4762838338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:48:22.204790 17477 raft_consensus.cc:1261] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991 [term 1 FOLLOWER]: Refusing update from remote peer 3ab36752164b41e282202c4762838338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:48:22.205909 17551 consensus_queue.cc:1035] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:48:22.206212 17547 consensus_queue.cc:1035] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:48:22.228222 17064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:48:22.228389 17064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:48:22.231110 17064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Sending DeleteTablet for 3 replicas of tablet cec0d938cd77492e8e136cd0848a97a2
I20260620 21:48:22.231977 17457 tablet_service.cc:1514] Processing DeleteTablet for tablet cec0d938cd77492e8e136cd0848a97a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:48:22 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:53912
I20260620 21:48:22.232038 17323 tablet_service.cc:1514] Processing DeleteTablet for tablet cec0d938cd77492e8e136cd0848a97a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:48:22 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:42894
I20260620 21:48:22.232571 17565 tablet_replica.cc:330] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d: stopping tablet replica
I20260620 21:48:22.232616 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 17126
I20260620 21:48:22.232980 17565 raft_consensus.cc:2229] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:48:22.233338 17565 raft_consensus.cc:2258] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:48:22.234701 17565 ts_tablet_manager.cc:1905] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:48:22.236927 17564 tablet_replica.cc:330] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991: stopping tablet replica
I20260620 21:48:22.237133 17565 ts_tablet_manager.cc:1918] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:48:22.237226 17565 log.cc:1196] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/cec0d938cd77492e8e136cd0848a97a2
I20260620 21:48:22.237391 17564 raft_consensus.cc:2229] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:48:22.237591 17565 ts_tablet_manager.cc:1939] T cec0d938cd77492e8e136cd0848a97a2 P 49f3320599cc4370b7ec5e84a692450d: Deleting consensus metadata
I20260620 21:48:22.237787 17564 raft_consensus.cc:2258] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:48:22.239042 17049 catalog_manager.cc:4841] TS 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845): tablet cec0d938cd77492e8e136cd0848a97a2 (table test-table [id=116eaa23a58443adb20a12d556cfdb52]) successfully deleted
I20260620 21:48:22.239369 17564 ts_tablet_manager.cc:1905] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 21:48:22.239758 17051 connection.cc:535] client connection to 127.1.155.129:36685 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 21:48:22.239910 17051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 21:48:22.240787 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 17260
I20260620 21:48:22.241196 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:22.240820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:36685 (local address 127.0.0.1:35786)
0620 21:48:22.241025 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:22.241105 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 21:48:22.242348 17564 ts_tablet_manager.cc:1918] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:48:22.242441 17564 log.cc:1196] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/cec0d938cd77492e8e136cd0848a97a2
I20260620 21:48:22.242799 17564 ts_tablet_manager.cc:1939] T cec0d938cd77492e8e136cd0848a97a2 P f29d3ae3dfc14c5282ecfc045bd73991: Deleting consensus metadata
W20260620 21:48:22.242842 17051 catalog_manager.cc:4584] TS 3ab36752164b41e282202c4762838338 (127.1.155.129:36685): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cec0d938cd77492e8e136cd0848a97a2: Network error: Client connection negotiation failed: client connection to 127.1.155.129:36685: connect: Connection refused (error 111)
I20260620 21:48:22.243956 17048 catalog_manager.cc:4841] TS f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431): tablet cec0d938cd77492e8e136cd0848a97a2 (table test-table [id=116eaa23a58443adb20a12d556cfdb52]) successfully deleted
I20260620 21:48:22.247398 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 17394
I20260620 21:48:22.252852 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 17032
I20260620 21:48:42.259588 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:36535
--webserver_interface=127.1.155.190
--webserver_port=34045
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:36535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:48:42.379380 17567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:42.379668 17567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:42.379709 17567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:42.383922 17567 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 21:48:42.383993 17567 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 21:48:42.384011 17567 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 21:48:42.384024 17567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:48:42.384042 17567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:42.384060 17567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:42.384111 17567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:48:42.384141 17567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:48:42.389715 17567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:36535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:36535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=34045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:42.390910 17567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:42.391993 17567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:42.405676 17574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:42.405676 17576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:42.405694 17573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:42.406203 17567 server_base.cc:1034] running on GCE node
I20260620 21:48:42.406769 17567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:42.407995 17567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:42.409184 17567 hybrid_clock.cc:648] HybridClock initialized: now 1781992122409166 us; error 32 us; skew 500 ppm
Jun 20 21:48:42 dist-test-slave-5bvn krb5kdc[17019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992122, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:42.416007 17567 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:48:42.416988 17567 webserver.cc:460] Webserver started at http://127.1.155.190:34045/ using document root <none> and password file <none>
I20260620 21:48:42.417558 17567 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:42.417629 17567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:42.421819 17567 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:48:42.424273 17583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:42.425575 17567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 21:48:42.425710 17567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "503b4dc73f344f549d96e55154f64c5c"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "83dbbccbd54a318b823008d995c84636"
server_key_iv: "55f2e34121d966247362c5f5bd0bcce3"
server_key_version: "encryptionkey@0"
I20260620 21:48:42.426400 17567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:42.441107 17567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:42.444474 17567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:42.444707 17567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:42.452661 17567 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:36535
I20260620 21:48:42.452966 17635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:36535 every 8 connection(s)
I20260620 21:48:42.453985 17567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:48:42.456764 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 17567
I20260620 21:48:42.457260 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:36685
--local_ip_for_outbound_sockets=127.1.155.129
--tserver_master_addrs=127.1.155.190:36535
--webserver_port=34557
--webserver_interface=127.1.155.129
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:48:42.459324 17636 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:48:42.461364 17636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Bootstrap starting.
I20260620 21:48:42.478999 17636 log.cc:824] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:42.489171 17636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:48:42.489667 17636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Bootstrap complete.
I20260620 21:48:42.494024 17636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } }
I20260620 21:48:42.494441 17636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 503b4dc73f344f549d96e55154f64c5c, State: Initialized, Role: FOLLOWER
I20260620 21:48:42.494961 17636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } }
I20260620 21:48:42.495078 17636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:48:42.495175 17636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:48:42.495291 17636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:48:42.497290 17636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } }
I20260620 21:48:42.497668 17636 leader_election.cc:302] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 503b4dc73f344f549d96e55154f64c5c; no voters:
I20260620 21:48:42.498051 17636 leader_election.cc:288] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 21:48:42.498169 17640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:48:42.498384 17640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 2 LEADER]: Becoming Leader. State: Replica: 503b4dc73f344f549d96e55154f64c5c, State: Running, Role: LEADER
I20260620 21:48:42.498750 17640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } }
I20260620 21:48:42.499394 17636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:48:42.500236 17641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "503b4dc73f344f549d96e55154f64c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } } }
I20260620 21:48:42.500368 17641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [sys.catalog]: This master's current role is: LEADER
I20260620 21:48:42.500803 17642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 503b4dc73f344f549d96e55154f64c5c. Latest consensus state: current_term: 2 leader_uuid: "503b4dc73f344f549d96e55154f64c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } } }
I20260620 21:48:42.500973 17642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [sys.catalog]: This master's current role is: LEADER
I20260620 21:48:42.501294 17649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:48:42.506660 17649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:48:42.507268 17649 catalog_manager.cc:1259] Loaded cluster ID: 9bdd1bb87aef45b1aef39ff06c61c214
I20260620 21:48:42.507329 17649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:48:42.510416 17649 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:48:42.511214 17649 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Loaded TSK: 0
I20260620 21:48:42.530853 17649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Generated new TSK 1
I20260620 21:48:42.531284 17649 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:48:42.581935 17638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:42.582195 17638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:42.582275 17638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:42.586369 17638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:42.586433 17638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:42.586519 17638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:48:42.591804 17638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:36685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=34557
--tserver_master_addrs=127.1.155.190:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:42.592839 17638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:42.593926 17638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:42.603566 17664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:42.603540 17663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:42.603533 17666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:42.604549 17638 server_base.cc:1034] running on GCE node
I20260620 21:48:42.605016 17638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:42.605798 17638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:42.607018 17638 hybrid_clock.cc:648] HybridClock initialized: now 1781992122607003 us; error 48 us; skew 500 ppm
Jun 20 21:48:42 dist-test-slave-5bvn krb5kdc[17019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992122, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:42.610435 17638 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:48:42.611457 17638 webserver.cc:460] Webserver started at http://127.1.155.129:34557/ using document root <none> and password file <none>
I20260620 21:48:42.612116 17638 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:42.612223 17638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:42.616328 17638 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:48:42.618805 17673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:42.620281 17638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:48:42.620424 17638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "3ab36752164b41e282202c4762838338"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "944670c29946f49eaac942bb5d01c5c5"
server_key_iv: "47b81451ac28f1bd6eec8d31543619f7"
server_key_version: "encryptionkey@0"
I20260620 21:48:42.620990 17638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:42.642215 17638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:42.645437 17638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:42.645658 17638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:42.646356 17638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:48:42.647514 17680 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 21:48:42.651842 17638 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 21:48:42.651945 17638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 21:48:42.652041 17638 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 21:48:42.653987 17638 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 21:48:42.654098 17638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260620 21:48:42.654779 17680 tablet_bootstrap.cc:492] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: Bootstrap starting.
I20260620 21:48:42.669049 17787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:36685 every 8 connection(s)
I20260620 21:48:42.669050 17638 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:36685
I20260620 21:48:42.670805 17638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:48:42.674979 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 17638
I20260620 21:48:42.675529 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:43845
--local_ip_for_outbound_sockets=127.1.155.130
--tserver_master_addrs=127.1.155.190:36535
--webserver_port=40841
--webserver_interface=127.1.155.130
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:48:42.678645 17680 log.cc:824] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:48:42 dist-test-slave-5bvn krb5kdc[17019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992122, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:48:42.688894 17680 tablet_bootstrap.cc:492] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:48:42.689453 17680 tablet_bootstrap.cc:492] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: Bootstrap complete.
I20260620 21:48:42.691321 17680 ts_tablet_manager.cc:1397] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: Time spent bootstrapping tablet: real 0.037s user 0.026s sys 0.008s
I20260620 21:48:42.694857 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:42.673237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:39343)
0620 21:48:42.674679 (+ 1442us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:42.674746 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:42.676094 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:42.677752 (+ 1658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:42.677761 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:42.678377 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:42.679246 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:42.679275 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:42.681332 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:42.681336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:42.682786 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:42.682798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:42.684087 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:42.684092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:42.684278 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:42.686008 (+ 1730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:42.686041 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:42.688004 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:42.691213 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:42.691220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:42.691234 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:42.691612 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:42.692072 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:42.692075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:42.692077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:42.692254 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:42.693025 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:42.693031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:42.693314 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 21:48:42.693501 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 21:48:42.693891 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1126,"thread_start_us":171,"threads_started":1}
I20260620 21:48:42.694464 17680 raft_consensus.cc:348] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } } peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } }
I20260620 21:48:42.694967 17680 raft_consensus.cc:729] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab36752164b41e282202c4762838338, State: Initialized, Role: FOLLOWER
I20260620 21:48:42.695449 17680 consensus_queue.cc:260] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } } peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } }
I20260620 21:48:42.696480 17680 ts_tablet_manager.cc:1428] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 21:48:42.696622 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:42.673229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:39343 (local address 127.1.155.190:36535)
0620 21:48:42.673685 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:42.673708 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:42.676448 (+ 2740us) server_negotiation.cc:407] Connection header received
0620 21:48:42.676772 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:42.676789 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:42.677274 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:42.677661 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:42.679474 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:42.681139 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:42.682944 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:42.683694 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:42.683863 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:42.688278 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:42.688285 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:42.688301 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:42.688337 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:42.690914 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:42.691780 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:42.691787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:42.691793 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:42.691954 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:42.692373 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:42.692376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:42.692378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:42.692843 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:42.692970 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:42.695474 (+ 2504us) server_negotiation.cc:299] Negotiation successful
0620 21:48:42.695798 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":145,"threads_started":1}
I20260620 21:48:42.699530 17788 heartbeater.cc:347] Connected to a master server at 127.1.155.190:36535
I20260620 21:48:42.699806 17788 heartbeater.cc:464] Registering TS with master...
I20260620 21:48:42.700379 17788 heartbeater.cc:511] Master 127.1.155.190:36535 requested a full tablet report, sending...
I20260620 21:48:42.703444 17600 ts_manager.cc:194] Registered new tserver with Master: 3ab36752164b41e282202c4762838338 (127.1.155.129:36685)
I20260620 21:48:42.707995 17600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:39343
I20260620 21:48:42.711035 17788 heartbeater.cc:503] Master 127.1.155.190:36535 was elected leader, sending a full tablet report...
Jun 20 21:48:42 dist-test-slave-5bvn krb5kdc[17019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992122, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for kudu/127.1.155.129@KRBTEST.COM
I20260620 21:48:42.720080 17795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:42.707293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:36685 (local address 127.0.0.1:33438)
0620 21:48:42.707662 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:42.707706 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:42.707830 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:42.709230 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:42.709236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:42.709655 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:42.709779 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:42.709788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:42.711293 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:42.711296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:42.712140 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:42.712150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:42.713127 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:42.713130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:42.713208 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:42.713807 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:42.713895 (+ 88us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:42.715794 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:42.718142 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:42.718152 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:42.718157 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:42.718445 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:42.718784 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:42.718787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:42.718790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:42.718875 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:42.719431 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:42.719437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:42.719554 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 21:48:42.719702 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 21:48:42.719898 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
I20260620 21:48:42.720780 17796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:42.708064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33438 (local address 127.1.155.129:36685)
0620 21:48:42.708377 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:42.708389 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:42.708411 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:48:42.708485 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:42.708490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:42.708688 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:42.709220 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:42.709947 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:42.711187 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:42.712287 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:42.712993 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:42.713093 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:42.715962 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:42.715972 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:42.715984 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:42.716011 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:42.718009 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:42.718573 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:42.718579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:42.718581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:42.718682 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:42.718969 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:42.718972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:42.718974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:42.719261 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:42.719378 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:42.720455 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0620 21:48:42.720629 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":117,"threads_started":1}
I20260620 21:48:42.722366 17704 tablet_service.cc:1514] Processing DeleteTablet for tablet cec0d938cd77492e8e136cd0848a97a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:48:22 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:33438
I20260620 21:48:42.723402 17797 tablet_replica.cc:330] stopping tablet replica
I20260620 21:48:42.723807 17797 raft_consensus.cc:2229] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:48:42.724116 17797 raft_consensus.cc:2258] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:48:42.725461 17797 ts_tablet_manager.cc:1905] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:48:42.729571 17797 ts_tablet_manager.cc:1918] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:48:42.729662 17797 log.cc:1196] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal/wals/cec0d938cd77492e8e136cd0848a97a2
I20260620 21:48:42.729980 17797 ts_tablet_manager.cc:1939] T cec0d938cd77492e8e136cd0848a97a2 P 3ab36752164b41e282202c4762838338: Deleting consensus metadata
I20260620 21:48:42.731685 17587 catalog_manager.cc:4841] TS 3ab36752164b41e282202c4762838338 (127.1.155.129:36685): tablet cec0d938cd77492e8e136cd0848a97a2 (table test-table [id=116eaa23a58443adb20a12d556cfdb52]) successfully deleted
W20260620 21:48:42.806437 17792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:42.806747 17792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:42.806915 17792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:42.811479 17792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:42.811585 17792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:42.811678 17792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:48:42.817196 17792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:43845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=40841
--tserver_master_addrs=127.1.155.190:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:42.818374 17792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:42.819447 17792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:42.829448 17803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:42.829454 17805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:42.829473 17802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:42.829675 17792 server_base.cc:1034] running on GCE node
I20260620 21:48:42.830302 17792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:42.830957 17792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:42.832159 17792 hybrid_clock.cc:648] HybridClock initialized: now 1781992122832134 us; error 63 us; skew 500 ppm
Jun 20 21:48:42 dist-test-slave-5bvn krb5kdc[17019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992122, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:42.835091 17792 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:48:42.836045 17792 webserver.cc:460] Webserver started at http://127.1.155.130:40841/ using document root <none> and password file <none>
I20260620 21:48:42.836614 17792 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:42.836740 17792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:42.840715 17792 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:48:42.842945 17812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:42.844316 17792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:48:42.844465 17792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "49f3320599cc4370b7ec5e84a692450d"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "412f77cc7d84bdde00bcd588a3b7483c"
server_key_iv: "624c008104d6013887d3d1fc875d5aa3"
server_key_version: "encryptionkey@0"
I20260620 21:48:42.845027 17792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:42.866971 17792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:42.870052 17792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:42.870270 17792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:42.870867 17792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:48:42.871747 17792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:48:42.871819 17792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:42.871892 17792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:48:42.871942 17792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:42.881103 17792 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:43845
I20260620 21:48:42.881129 17925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:43845 every 8 connection(s)
I20260620 21:48:42.882133 17792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:48:42.884034 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 17792
I20260620 21:48:42.884509 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:43431
--local_ip_for_outbound_sockets=127.1.155.131
--tserver_master_addrs=127.1.155.190:36535
--webserver_port=33817
--webserver_interface=127.1.155.131
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:48:42 dist-test-slave-5bvn krb5kdc[17019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992122, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:48:42.897590 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:42.884070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:35049 (local address 127.1.155.190:36535)
0620 21:48:42.884243 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:42.884247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:42.885271 (+ 1024us) server_negotiation.cc:407] Connection header received
0620 21:48:42.886588 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:42.886594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:42.886669 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:42.886803 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:42.888023 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:42.888951 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:42.890266 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:42.890715 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:42.890852 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:42.893390 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:42.893396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:42.893402 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:42.893446 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:42.895437 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:42.895965 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:42.895969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:42.895971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:42.896062 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:42.896455 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:42.896458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:42.896460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:42.896677 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:42.896773 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:42.897258 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 21:48:42.897404 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 21:48:42.898470 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:42.884486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:35049)
0620 21:48:42.885041 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:42.885103 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:42.886367 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:42.886990 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:42.886997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:42.887375 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:42.887859 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:42.887872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:42.889098 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:42.889101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:42.890083 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:42.890095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:42.890882 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:42.890884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:42.891038 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:42.891711 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:42.891727 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:42.893210 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:42.895574 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:42.895580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:42.895593 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:42.895864 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:42.896212 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:42.896215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:42.896216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:42.896357 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:42.896782 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:42.896786 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:42.897020 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 21:48:42.897189 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 21:48:42.897444 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":186,"threads_started":1}
I20260620 21:48:42.899714 17926 heartbeater.cc:347] Connected to a master server at 127.1.155.190:36535
I20260620 21:48:42.899972 17926 heartbeater.cc:464] Registering TS with master...
I20260620 21:48:42.900508 17926 heartbeater.cc:511] Master 127.1.155.190:36535 requested a full tablet report, sending...
I20260620 21:48:42.901602 17600 ts_manager.cc:194] Registered new tserver with Master: 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845)
I20260620 21:48:42.902245 17600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:35049
W20260620 21:48:42.998744 17929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:48:42.999037 17929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:48:42.999116 17929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:48:43.004230 17929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:48:43.004328 17929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:48:43.004431 17929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:48:43.010643 17929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:43431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=33817
--tserver_master_addrs=127.1.155.190:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:48:43.011842 17929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:48:43.012992 17929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:48:43.023716 17934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:48:43.023761 17937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:43.023975 17929 server_base.cc:1034] running on GCE node
W20260620 21:48:43.024120 17935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:48:43.024632 17929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:48:43.025390 17929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:48:43.026646 17929 hybrid_clock.cc:648] HybridClock initialized: now 1781992123026625 us; error 33 us; skew 500 ppm
Jun 20 21:48:43 dist-test-slave-5bvn krb5kdc[17019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992123, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:48:43.029538 17929 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:48:43.030539 17929 webserver.cc:460] Webserver started at http://127.1.155.131:33817/ using document root <none> and password file <none>
I20260620 21:48:43.031232 17929 fs_manager.cc:359] Metadata directory not provided
I20260620 21:48:43.031322 17929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:48:43.035218 17929 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:48:43.037395 17944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:43.038928 17929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:48:43.039031 17929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "f29d3ae3dfc14c5282ecfc045bd73991"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "9d674599504f5abd7c56b9d4c8335a32"
server_key_iv: "6409b999d03d7b27b03c80cafaa3c25f"
server_key_version: "encryptionkey@0"
I20260620 21:48:43.039556 17929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:48:43.058786 17929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:48:43.062055 17929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:48:43.062232 17929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:48:43.062764 17929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:48:43.063537 17929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:48:43.063583 17929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:43.063623 17929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:48:43.063644 17929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:48:43.072405 17929 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:43431
I20260620 21:48:43.072422 18057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:43431 every 8 connection(s)
I20260620 21:48:43.073372 17929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:48:43.081724 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 17929
Jun 20 21:48:43 dist-test-slave-5bvn krb5kdc[17019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992123, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:48:43.087141 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:43.075223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:34193 (local address 127.1.155.190:36535)
0620 21:48:43.075359 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:43.075363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:43.076185 (+ 822us) server_negotiation.cc:407] Connection header received
0620 21:48:43.077504 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:43.077507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:43.077570 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:43.077680 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:43.078868 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.079395 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.080517 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.080742 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.080851 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:43.083572 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:43.083576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:43.083578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:43.083606 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:43.085022 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:43.085552 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:43.085555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:43.085557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:43.085635 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:43.086038 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:43.086041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:43.086043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:43.086226 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:43.086325 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:43.086840 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 21:48:43.086961 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:48:43.088165 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:43.075578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:34193)
0620 21:48:43.076023 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:43.076064 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:43.077289 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:43.077879 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:43.077886 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:43.078241 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:48:43.078717 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:43.078731 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.079520 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.079524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:43.080391 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:43.080400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.080891 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.080895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:43.081063 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:43.081721 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:48:43.081737 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:48:43.083404 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:48:43.085158 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:43.085165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:43.085180 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:43.085448 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:43.085733 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:48:43.085738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:48:43.085740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:48:43.085935 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:48:43.086336 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:48:43.086344 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:48:43.086594 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 21:48:43.086779 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 21:48:43.087108 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":106,"threads_started":1}
I20260620 21:48:43.089303 18058 heartbeater.cc:347] Connected to a master server at 127.1.155.190:36535
I20260620 21:48:43.089531 18058 heartbeater.cc:464] Registering TS with master...
I20260620 21:48:43.090126 18058 heartbeater.cc:511] Master 127.1.155.190:36535 requested a full tablet report, sending...
I20260620 21:48:43.090197 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:43.082502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.1.155.190:36535)
0620 21:48:43.082767 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:43.082772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:43.082881 (+ 109us) server_negotiation.cc:407] Connection header received
0620 21:48:43.083040 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:43.083046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:43.083095 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:43.083196 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:43.083947 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.084458 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.085316 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.085543 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.085648 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:43.086998 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:43.087001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:43.087004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:43.087031 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:43.088577 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:43.089045 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:43.089051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:43.089052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:43.089129 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:43.089400 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:43.089405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:43.089407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:43.089568 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:43.089762 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:43.089962 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 21:48:43.090057 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
I20260620 21:48:43.091212 17600 ts_manager.cc:194] Registered new tserver with Master: f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431)
I20260620 21:48:43.091861 17600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:34193
I20260620 21:48:43.091979 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:48:43.095881 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:43.092576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.1.155.190:36535)
0620 21:48:43.092707 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:43.092711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:43.092900 (+ 189us) server_negotiation.cc:407] Connection header received
0620 21:48:43.093041 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:43.093043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:43.093123 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:43.093207 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:48:43.093429 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.093987 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.094829 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.095054 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.095139 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:43.095435 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:48:43.095565 (+ 130us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 21:48:43.095791 (+ 226us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37374: token expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 21:48:43.095980 18062 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37374: token expired
W20260620 21:48:43.096386 17528 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.155.190:36535)
I20260620 21:48:43.103600 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:43.096733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.1.155.190:36535)
0620 21:48:43.096858 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:43.096861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:43.096873 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:48:43.096954 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:43.096955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:43.096987 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:43.097044 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:48:43.097582 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.098102 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.098836 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.099096 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.099168 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:43.100599 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:48:43.100602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:48:43.100604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:48:43.100625 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:48:43.102144 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:43.102541 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:43.102544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:43.102546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:43.102621 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:48:43.102844 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:48:43.102846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:48:43.102847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:48:43.102999 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:48:43.103095 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:43.103309 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 21:48:43.103444 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:48:43.106137 17600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:48:43.109431 17600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:48:43.117185 17704 tablet_service.cc:1467] Processing CreateTablet for tablet 893c4abc4aa745b0a8218aaff4d0963c (DEFAULT_TABLE table=test-table [id=3beef5f069154cb887a3f955302f5d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:48:43.117928 17704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 893c4abc4aa745b0a8218aaff4d0963c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:48:43.120886 17680 tablet_bootstrap.cc:492] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338: Bootstrap starting.
I20260620 21:48:43.121883 17680 tablet_bootstrap.cc:654] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338: Neither blocks nor log segments found. Creating new log.
I20260620 21:48:43.122922 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:43.116679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.1.155.131:43431)
0620 21:48:43.117238 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:43.117252 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:43.117275 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:48:43.117352 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:43.117357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:43.117583 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:43.117741 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:43.118260 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.119428 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.121207 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.122315 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.122443 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:43.122598 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:43.122699 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 21:48:43.122796 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":163,"threads_started":1}
I20260620 21:48:43.123027 17795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:43.116269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:43845 (local address 127.0.0.1:51330)
0620 21:48:43.116444 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:43.116460 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:43.116545 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:43.117737 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:43.117740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:43.117757 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:43.118100 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:43.118114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.119527 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.119533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:43.121059 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:43.121071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.122517 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.122521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:43.122644 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:43.122671 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:48:43.122728 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 21:48:43.122821 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":108}
I20260620 21:48:43.122908 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:43.116430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:43431 (local address 127.0.0.1:51842)
0620 21:48:43.116756 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:43.116770 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:43.116863 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:43.117958 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:43.117961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:43.117977 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:43.118126 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:43.118135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.119569 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.119572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:43.121062 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:43.121071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.122440 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.122444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:43.122542 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:43.122593 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 21:48:43.122672 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 21:48:43.122758 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":173,"thread_start_us":96,"threads_started":1}
I20260620 21:48:43.123301 17680 tablet_bootstrap.cc:492] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338: No bootstrap required, opened a new log
I20260620 21:48:43.123396 17680 ts_tablet_manager.cc:1397] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 21:48:43.123713 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:43.116727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.1.155.130:43845)
0620 21:48:43.117241 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:43.117252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:43.117275 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:48:43.117352 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:43.117357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:43.117575 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:43.117741 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:43.118365 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.119420 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.121447 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.122368 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.123279 (+ 911us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:43.123429 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:43.123482 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 21:48:43.123582 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":166,"threads_started":1}
I20260620 21:48:43.123868 17680 raft_consensus.cc:348] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:48:43.124008 17680 raft_consensus.cc:374] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:48:43.124055 17680 raft_consensus.cc:729] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab36752164b41e282202c4762838338, State: Initialized, Role: FOLLOWER
I20260620 21:48:43.124182 17680 consensus_queue.cc:260] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:48:43.124491 17680 ts_tablet_manager.cc:1428] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260620 21:48:43.126556 17860 tablet_service.cc:1467] Processing CreateTablet for tablet 893c4abc4aa745b0a8218aaff4d0963c (DEFAULT_TABLE table=test-table [id=3beef5f069154cb887a3f955302f5d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:48:43.126592 17992 tablet_service.cc:1467] Processing CreateTablet for tablet 893c4abc4aa745b0a8218aaff4d0963c (DEFAULT_TABLE table=test-table [id=3beef5f069154cb887a3f955302f5d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:48:43.127575 17860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 893c4abc4aa745b0a8218aaff4d0963c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:48:43.127660 17992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 893c4abc4aa745b0a8218aaff4d0963c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:48:43.134878 18072 tablet_bootstrap.cc:492] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d: Bootstrap starting.
I20260620 21:48:43.135846 18071 tablet_bootstrap.cc:492] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991: Bootstrap starting.
I20260620 21:48:43.137715 18072 tablet_bootstrap.cc:654] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d: Neither blocks nor log segments found. Creating new log.
I20260620 21:48:43.138397 18071 tablet_bootstrap.cc:654] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991: Neither blocks nor log segments found. Creating new log.
I20260620 21:48:43.138761 18072 log.cc:824] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:43.139237 18071 log.cc:824] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991: Log is configured to *not* fsync() on all Append() calls
I20260620 21:48:43.140920 18072 tablet_bootstrap.cc:492] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d: No bootstrap required, opened a new log
I20260620 21:48:43.141072 18071 tablet_bootstrap.cc:492] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991: No bootstrap required, opened a new log
I20260620 21:48:43.141119 18072 ts_tablet_manager.cc:1397] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:48:43.141266 18071 ts_tablet_manager.cc:1397] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 21:48:43.144191 18072 raft_consensus.cc:348] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:48:43.144191 18071 raft_consensus.cc:348] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:48:43.144419 18072 raft_consensus.cc:374] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:48:43.144464 18071 raft_consensus.cc:374] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:48:43.144495 18072 raft_consensus.cc:729] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f3320599cc4370b7ec5e84a692450d, State: Initialized, Role: FOLLOWER
I20260620 21:48:43.144543 18071 raft_consensus.cc:729] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29d3ae3dfc14c5282ecfc045bd73991, State: Initialized, Role: FOLLOWER
I20260620 21:48:43.144944 18072 consensus_queue.cc:260] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:48:43.144944 18071 consensus_queue.cc:260] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:48:43.145561 17926 heartbeater.cc:503] Master 127.1.155.190:36535 was elected leader, sending a full tablet report...
I20260620 21:48:43.145560 18058 heartbeater.cc:503] Master 127.1.155.190:36535 was elected leader, sending a full tablet report...
I20260620 21:48:43.145949 18071 ts_tablet_manager.cc:1428] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 21:48:43.145949 18072 ts_tablet_manager.cc:1428] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 21:48:43.172191 17789 tablet.cc:2369] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:48:43.191524 18075 raft_consensus.cc:482] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:48:43.191771 18075 raft_consensus.cc:504] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:48:43.192960 18075 leader_election.cc:288] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431), 3ab36752164b41e282202c4762838338 (127.1.155.129:36685)
I20260620 21:48:43.197295 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:43.193244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:43431 (local address 127.1.155.130:60471)
0620 21:48:43.193513 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:43.193528 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:43.193648 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:43.194005 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:43.194007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:43.194021 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:43.194155 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:43.194162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.195029 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.195031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:43.196116 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:43.196126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.196858 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.196860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:43.196972 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:43.196999 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:48:43.197075 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 21:48:43.197168 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260620 21:48:43.197322 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:43.193377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:60471 (local address 127.1.155.131:43431)
0620 21:48:43.193522 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:43.193528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:43.193621 (+ 93us) server_negotiation.cc:407] Connection header received
0620 21:48:43.193769 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:43.193773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:43.193852 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:43.193959 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:43.194279 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.194923 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.196242 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.196748 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.196866 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:43.196917 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:43.197105 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 21:48:43.197179 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:48:43.197971 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:43.193244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:36685 (local address 127.1.155.130:47701)
0620 21:48:43.193724 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:43.193739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:48:43.193868 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:48:43.194187 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:48:43.194190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:48:43.194212 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:48:43.194660 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:43.194667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.195495 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.195498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:43.196489 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:48:43.196496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.197571 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.197575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:48:43.197663 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:43.197683 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 21:48:43.197752 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 21:48:43.197817 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":99,"thread_start_us":133,"threads_started":1}
I20260620 21:48:43.197989 17796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:43.193429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:47701 (local address 127.1.155.129:36685)
0620 21:48:43.193567 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:43.193572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:43.193862 (+ 290us) server_negotiation.cc:407] Connection header received
0620 21:48:43.193984 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:43.193990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:43.194054 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:43.194154 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:48:43.194802 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.195392 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.196602 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.197252 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.197374 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:43.197524 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:43.197771 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 21:48:43.197883 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:48:43.197992 18012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893c4abc4aa745b0a8218aaff4d0963c" candidate_uuid: "49f3320599cc4370b7ec5e84a692450d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" is_pre_election: true
I20260620 21:48:43.198334 18012 raft_consensus.cc:2454] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49f3320599cc4370b7ec5e84a692450d in term 0.
I20260620 21:48:43.198616 17733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893c4abc4aa745b0a8218aaff4d0963c" candidate_uuid: "49f3320599cc4370b7ec5e84a692450d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab36752164b41e282202c4762838338" is_pre_election: true
I20260620 21:48:43.198829 17813 leader_election.cc:302] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f3320599cc4370b7ec5e84a692450d, f29d3ae3dfc14c5282ecfc045bd73991; no voters:
I20260620 21:48:43.198863 17733 raft_consensus.cc:2454] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49f3320599cc4370b7ec5e84a692450d in term 0.
I20260620 21:48:43.199091 18075 raft_consensus.cc:2781] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:48:43.199193 18075 raft_consensus.cc:482] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:48:43.199249 18075 raft_consensus.cc:3037] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:48:43.200628 18075 raft_consensus.cc:504] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:48:43.200994 18075 leader_election.cc:288] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [CANDIDATE]: Term 1 election: Requested vote from peers f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431), 3ab36752164b41e282202c4762838338 (127.1.155.129:36685)
I20260620 21:48:43.201372 18012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893c4abc4aa745b0a8218aaff4d0963c" candidate_uuid: "49f3320599cc4370b7ec5e84a692450d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29d3ae3dfc14c5282ecfc045bd73991"
I20260620 21:48:43.201453 17733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893c4abc4aa745b0a8218aaff4d0963c" candidate_uuid: "49f3320599cc4370b7ec5e84a692450d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab36752164b41e282202c4762838338"
I20260620 21:48:43.201519 18012 raft_consensus.cc:3037] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:48:43.201610 17733 raft_consensus.cc:3037] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:48:43.202641 17733 raft_consensus.cc:2454] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49f3320599cc4370b7ec5e84a692450d in term 1.
I20260620 21:48:43.202658 18012 raft_consensus.cc:2454] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49f3320599cc4370b7ec5e84a692450d in term 1.
I20260620 21:48:43.202952 17813 leader_election.cc:302] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f3320599cc4370b7ec5e84a692450d, f29d3ae3dfc14c5282ecfc045bd73991; no voters:
I20260620 21:48:43.203130 18075 raft_consensus.cc:2781] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:48:43.203398 18075 raft_consensus.cc:686] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [term 1 LEADER]: Becoming Leader. State: Replica: 49f3320599cc4370b7ec5e84a692450d, State: Running, Role: LEADER
I20260620 21:48:43.203745 18075 consensus_queue.cc:237] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:48:43.206853 17600 catalog_manager.cc:5495] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d reported cstate change: term changed from 0 to 1, leader changed from <none> to 49f3320599cc4370b7ec5e84a692450d (127.1.155.130). New cstate: current_term: 1 leader_uuid: "49f3320599cc4370b7ec5e84a692450d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } health_report { overall_health: UNKNOWN } } }
I20260620 21:48:43.221980 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:43.218470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.1.155.130:43845)
0620 21:48:43.218596 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 21:48:43.218600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:48:43.218613 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:48:43.218685 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:48:43.218687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:48:43.218738 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:48:43.218826 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:48:43.219074 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.219621 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.220526 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:48:43.220747 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:48:43.220836 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:48:43.221090 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:48:43.221506 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:48:43.221599 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 21:48:43.221708 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 21:48:43.221760 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:48:43.227795 18012 raft_consensus.cc:1261] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991 [term 1 FOLLOWER]: Refusing update from remote peer 49f3320599cc4370b7ec5e84a692450d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:48:43.227797 17733 raft_consensus.cc:1261] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Refusing update from remote peer 49f3320599cc4370b7ec5e84a692450d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:48:43.228538 18075 consensus_queue.cc:1035] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:48:43.228742 18078 consensus_queue.cc:1035] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:48:43.234884 18082 mvcc.cc:205] Tried to move back new op lower bound from 7299039736733130752 to 7299039736644354048. Current Snapshot: MvccSnapshot[applied={T|T < 7299039736733130752}]
I20260620 21:48:43.234934 18081 mvcc.cc:205] Tried to move back new op lower bound from 7299039736733130752 to 7299039736644354048. Current Snapshot: MvccSnapshot[applied={T|T < 7299039736733130752}]
I20260620 21:48:43.243912 17600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:48:43.244108 17600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:48:43.246367 17600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Sending DeleteTablet for 3 replicas of tablet 893c4abc4aa745b0a8218aaff4d0963c
I20260620 21:48:43.247107 17704 tablet_service.cc:1514] Processing DeleteTablet for tablet 893c4abc4aa745b0a8218aaff4d0963c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:48:43 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:33438
I20260620 21:48:43.247187 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 17638
I20260620 21:48:43.247200 17860 tablet_service.cc:1514] Processing DeleteTablet for tablet 893c4abc4aa745b0a8218aaff4d0963c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:48:43 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:51330
I20260620 21:48:43.247987 17992 tablet_service.cc:1514] Processing DeleteTablet for tablet 893c4abc4aa745b0a8218aaff4d0963c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:48:43 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:51842
I20260620 21:48:43.248677 18091 tablet_replica.cc:330] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d: stopping tablet replica
I20260620 21:48:43.249050 18091 raft_consensus.cc:2229] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:48:43.249502 18091 raft_consensus.cc:2258] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:48:43.250254 18092 tablet_replica.cc:330] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991: stopping tablet replica
I20260620 21:48:43.250540 18092 raft_consensus.cc:2229] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:48:43.250840 18092 raft_consensus.cc:2258] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:48:43.251152 18091 ts_tablet_manager.cc:1905] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:48:43.252357 18092 ts_tablet_manager.cc:1905] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:48:43.253388 18091 ts_tablet_manager.cc:1918] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:48:43.253494 18091 log.cc:1196] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/893c4abc4aa745b0a8218aaff4d0963c
I20260620 21:48:43.253782 18091 ts_tablet_manager.cc:1939] T 893c4abc4aa745b0a8218aaff4d0963c P 49f3320599cc4370b7ec5e84a692450d: Deleting consensus metadata
W20260620 21:48:43.254498 17587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.155.129:36685 (error 108)
I20260620 21:48:43.255016 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 17792
I20260620 21:48:43.254985 17585 catalog_manager.cc:4841] TS 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845): tablet 893c4abc4aa745b0a8218aaff4d0963c (table test-table [id=3beef5f069154cb887a3f955302f5d3c]) successfully deleted
I20260620 21:48:43.256330 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:48:43.255984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:36685 (local address 127.0.0.1:33446)
0620 21:48:43.256133 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 21:48:43.256218 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 21:48:43.256580 18092 ts_tablet_manager.cc:1918] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:48:43.256682 18092 log.cc:1196] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/893c4abc4aa745b0a8218aaff4d0963c
W20260620 21:48:43.256547 17587 catalog_manager.cc:4584] TS 3ab36752164b41e282202c4762838338 (127.1.155.129:36685): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 893c4abc4aa745b0a8218aaff4d0963c: Network error: Client connection negotiation failed: client connection to 127.1.155.129:36685: connect: Connection refused (error 111)
I20260620 21:48:43.257037 18092 ts_tablet_manager.cc:1939] T 893c4abc4aa745b0a8218aaff4d0963c P f29d3ae3dfc14c5282ecfc045bd73991: Deleting consensus metadata
I20260620 21:48:43.258230 17584 catalog_manager.cc:4841] TS f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431): tablet 893c4abc4aa745b0a8218aaff4d0963c (table test-table [id=3beef5f069154cb887a3f955302f5d3c]) successfully deleted
I20260620 21:48:43.261933 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 17929
I20260620 21:48:43.267455 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 17567
I20260620 21:49:03.274082 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:36535
--webserver_interface=127.1.155.190
--webserver_port=34045
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:36535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:03.386313 18094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:03.386621 18094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:03.386698 18094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:03.391040 18094 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 21:49:03.391149 18094 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 21:49:03.391185 18094 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 21:49:03.391218 18094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:03.391263 18094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:03.391304 18094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:03.391342 18094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:03.391378 18094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:03.397112 18094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:36535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:36535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=34045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:03.398209 18094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:03.399286 18094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:03.408490 18102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:03.408469 18100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:03.408581 18099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:03.408594 18094 server_base.cc:1034] running on GCE node
I20260620 21:49:03.409341 18094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:03.410460 18094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:03.411679 18094 hybrid_clock.cc:648] HybridClock initialized: now 1781992143411635 us; error 47 us; skew 500 ppm
Jun 20 21:49:03 dist-test-slave-5bvn krb5kdc[17019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992143, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:03.414510 18094 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:49:03.415395 18094 webserver.cc:460] Webserver started at http://127.1.155.190:34045/ using document root <none> and password file <none>
I20260620 21:49:03.415930 18094 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:03.416018 18094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:03.419843 18094 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:49:03.422207 18109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:03.423483 18094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:49:03.423643 18094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "503b4dc73f344f549d96e55154f64c5c"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "83dbbccbd54a318b823008d995c84636"
server_key_iv: "55f2e34121d966247362c5f5bd0bcce3"
server_key_version: "encryptionkey@0"
I20260620 21:49:03.424187 18094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:03.462669 18094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:03.466005 18094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:03.466259 18094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:03.473755 18094 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:36535
I20260620 21:49:03.474076 18161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:36535 every 8 connection(s)
I20260620 21:49:03.474920 18094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:03.479265 18162 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:49:03.480860 18162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Bootstrap starting.
I20260620 21:49:03.481293 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 18094
I20260620 21:49:03.481771 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:36685
--local_ip_for_outbound_sockets=127.1.155.129
--tserver_master_addrs=127.1.155.190:36535
--webserver_port=34557
--webserver_interface=127.1.155.129
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:49:03.499039 18162 log.cc:824] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:03.510035 18162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:49:03.510448 18162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Bootstrap complete.
I20260620 21:49:03.514513 18162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } }
I20260620 21:49:03.514894 18162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 503b4dc73f344f549d96e55154f64c5c, State: Initialized, Role: FOLLOWER
I20260620 21:49:03.515358 18162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } }
I20260620 21:49:03.515478 18162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:03.515568 18162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:03.515683 18162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 2 FOLLOWER]: Advancing to term 3
I20260620 21:49:03.517789 18162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } }
I20260620 21:49:03.518227 18162 leader_election.cc:302] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 503b4dc73f344f549d96e55154f64c5c; no voters:
I20260620 21:49:03.518946 18162 leader_election.cc:288] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 21:49:03.519033 18166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 3 FOLLOWER]: Leader election won for term 3
I20260620 21:49:03.519263 18166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [term 3 LEADER]: Becoming Leader. State: Replica: 503b4dc73f344f549d96e55154f64c5c, State: Running, Role: LEADER
I20260620 21:49:03.519554 18166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } }
I20260620 21:49:03.520392 18162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:03.520854 18167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 503b4dc73f344f549d96e55154f64c5c. Latest consensus state: current_term: 3 leader_uuid: "503b4dc73f344f549d96e55154f64c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } } }
I20260620 21:49:03.520977 18167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:03.521278 18168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "503b4dc73f344f549d96e55154f64c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503b4dc73f344f549d96e55154f64c5c" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36535 } } }
I20260620 21:49:03.521356 18168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:03.521384 18171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:03.526480 18171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:03.527222 18171 catalog_manager.cc:1259] Loaded cluster ID: 9bdd1bb87aef45b1aef39ff06c61c214
I20260620 21:49:03.527287 18171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:03.529982 18171 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:03.530772 18171 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Loaded TSK: 0
I20260620 21:49:03.530861 18171 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 503b4dc73f344f549d96e55154f64c5c: Loaded TSK: 1
F20260620 21:49:03.532068 18171 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f82c72669ad google::LogMessage::Fail() at ??:0
@ 0x7f82c726988a google::LogMessage::SendToLog() at ??:0
@ 0x7f82c72664de google::LogMessage::Flush() at ??:0
@ 0x7f82c7267a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f82c9f2a18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f82c9f22f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f82d1bcb7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f82d1ba599c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f82d1ba6178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f82d1be7df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f82d1c0682b std::function<>::operator()() at ??:0
@ 0x7f82d1ba5b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f82d1ba6f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f82d1ba3579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f82d1be7640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f82c94b83d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f82c94b8c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f82c94ba3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f82c94a9512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f82caaed6db start_thread at ??:0
@ 0x7f82c5d6461f clone at ??:0
W20260620 21:49:03.616441 18164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:03.616812 18164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:03.616885 18164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:03.623646 18164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:03.623710 18164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:03.623812 18164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:49:03.631986 18164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:36685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=34557
--tserver_master_addrs=127.1.155.190:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:03.633392 18164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:03.634706 18164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:03.645574 18209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:03.645608 18212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:03.645578 18210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:03.646273 18164 server_base.cc:1034] running on GCE node
I20260620 21:49:03.646656 18164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:03.647279 18164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:03.648461 18164 hybrid_clock.cc:648] HybridClock initialized: now 1781992143648434 us; error 43 us; skew 500 ppm
Jun 20 21:49:03 dist-test-slave-5bvn krb5kdc[17019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992143, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:03.651352 18164 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:49:03.652194 18164 webserver.cc:460] Webserver started at http://127.1.155.129:34557/ using document root <none> and password file <none>
I20260620 21:49:03.652761 18164 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:03.652853 18164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:03.656828 18164 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:49:03.659173 18219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:03.660508 18164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:49:03.660665 18164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "3ab36752164b41e282202c4762838338"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "944670c29946f49eaac942bb5d01c5c5"
server_key_iv: "47b81451ac28f1bd6eec8d31543619f7"
server_key_version: "encryptionkey@0"
I20260620 21:49:03.661221 18164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:03.674072 18164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:03.677362 18164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:03.677583 18164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:03.678334 18164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:03.679494 18226 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 21:49:03.684779 18164 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 21:49:03.684878 18164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 21:49:03.684969 18164 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 21:49:03.686754 18164 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 21:49:03.686833 18164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 21:49:03.687106 18226 tablet_bootstrap.cc:492] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338: Bootstrap starting.
I20260620 21:49:03.698971 18164 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:36685
I20260620 21:49:03.699324 18333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:36685 every 8 connection(s)
I20260620 21:49:03.700122 18164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:49:03.708541 18226 log.cc:824] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:03.708945 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 18164
I20260620 21:49:03.709527 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:43845
--local_ip_for_outbound_sockets=127.1.155.130
--tserver_master_addrs=127.1.155.190:36535
--webserver_port=40841
--webserver_interface=127.1.155.130
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:49:03.717420 18226 tablet_bootstrap.cc:492] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:49:03.717890 18226 tablet_bootstrap.cc:492] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338: Bootstrap complete.
I20260620 21:49:03.719343 18226 ts_tablet_manager.cc:1397] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.016s
I20260620 21:49:03.722627 18226 raft_consensus.cc:348] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:49:03.723158 18226 raft_consensus.cc:729] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab36752164b41e282202c4762838338, State: Initialized, Role: FOLLOWER
I20260620 21:49:03.723668 18226 consensus_queue.cc:260] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:49:03.724735 18226 ts_tablet_manager.cc:1428] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260620 21:49:03.774266 18336 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:49:03.702317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:44495)
0620 21:49:03.702841 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:03.702899 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:03.703805 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:03.773264 (+ 69459us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":156,"threads_started":1}
W20260620 21:49:03.774852 18334 heartbeater.cc:644] Failed to heartbeat to 127.1.155.190:36535 (0 consecutive failures): Network error: Failed to ping master at 127.1.155.190:36535: Client connection negotiation failed: client connection to 127.1.155.190:36535: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 21:49:03.775628 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:03.775347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:40321)
0620 21:49:03.775491 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:03.775563 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 21:49:03.776328 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:03.776132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:46151)
0620 21:49:03.776238 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:03.776281 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 21:49:03.776593 18334 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:49:03.824586 18338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:03.824905 18338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:03.824983 18338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:03.829088 18338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:03.829183 18338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:03.829291 18338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:49:03.834537 18338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:43845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=40841
--tserver_master_addrs=127.1.155.190:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:03.835587 18338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:03.836596 18338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:03.846109 18344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:03.846109 18345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:03.846143 18347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:03.846659 18338 server_base.cc:1034] running on GCE node
I20260620 21:49:03.847054 18338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:03.847671 18338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:03.848865 18338 hybrid_clock.cc:648] HybridClock initialized: now 1781992143848841 us; error 32 us; skew 500 ppm
Jun 20 21:49:03 dist-test-slave-5bvn krb5kdc[17019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992143, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:03.851627 18338 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:49:03.852533 18338 webserver.cc:460] Webserver started at http://127.1.155.130:40841/ using document root <none> and password file <none>
I20260620 21:49:03.853111 18338 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:03.853173 18338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:03.856940 18338 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:49:03.858979 18354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:03.860172 18338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:49:03.860292 18338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "49f3320599cc4370b7ec5e84a692450d"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "412f77cc7d84bdde00bcd588a3b7483c"
server_key_iv: "624c008104d6013887d3d1fc875d5aa3"
server_key_version: "encryptionkey@0"
I20260620 21:49:03.860821 18338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:03.890951 18338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:03.894081 18338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:03.894291 18338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:03.894902 18338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:03.895717 18338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:03.895763 18338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:03.895802 18338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:03.895895 18338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:03.905185 18338 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:43845
I20260620 21:49:03.905206 18467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:43845 every 8 connection(s)
I20260620 21:49:03.906239 18338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:49:03.907033 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 18338
I20260620 21:49:03.907513 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:43431
--local_ip_for_outbound_sockets=127.1.155.131
--tserver_master_addrs=127.1.155.190:36535
--webserver_port=33817
--webserver_interface=127.1.155.131
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:49:03.910847 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:03.908677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:48131)
0620 21:49:03.909295 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:03.909921 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":146,"threads_started":1}
W20260620 21:49:03.911338 18468 heartbeater.cc:644] Failed to heartbeat to 127.1.155.190:36535 (0 consecutive failures): Network error: Failed to ping master at 127.1.155.190:36535: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
I20260620 21:49:03.912079 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:03.911867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:54479)
0620 21:49:03.911981 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:03.912025 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 21:49:03.912777 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:03.912508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:35529)
0620 21:49:03.912634 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:03.912694 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 21:49:03.912968 18468 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:49:04.018033 18470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:04.018311 18470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:04.018357 18470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:04.022506 18470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:04.022569 18470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:04.022665 18470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:49:04.027926 18470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:43431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=33817
--tserver_master_addrs=127.1.155.190:36535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:04.029088 18470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:04.030200 18470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:04.040019 18479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:04.040064 18476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:04.040066 18477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:04.040179 18470 server_base.cc:1034] running on GCE node
I20260620 21:49:04.040772 18470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:04.041442 18470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:04.042637 18470 hybrid_clock.cc:648] HybridClock initialized: now 1781992144042622 us; error 25 us; skew 500 ppm
Jun 20 21:49:04 dist-test-slave-5bvn krb5kdc[17019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992144, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:04.045367 18470 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:49:04.046332 18470 webserver.cc:460] Webserver started at http://127.1.155.131:33817/ using document root <none> and password file <none>
I20260620 21:49:04.046907 18470 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:04.047003 18470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:04.050675 18470 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:49:04.052764 18486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:04.054100 18470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:49:04.054234 18470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "f29d3ae3dfc14c5282ecfc045bd73991"
format_stamp: "Formatted at 2026-06-20 21:48:21 on dist-test-slave-5bvn"
server_key: "9d674599504f5abd7c56b9d4c8335a32"
server_key_iv: "6409b999d03d7b27b03c80cafaa3c25f"
server_key_version: "encryptionkey@0"
I20260620 21:49:04.054796 18470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:04.080695 18470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:04.083968 18470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:04.084195 18470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:04.084810 18470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:04.085659 18470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:04.085734 18470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:04.085807 18470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:04.085876 18470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:04.095259 18470 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:43431
I20260620 21:49:04.095343 18599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:43431 every 8 connection(s)
I20260620 21:49:04.096297 18470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:49:04.100044 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:04.098457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:35791)
0620 21:49:04.098920 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:04.099366 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
W20260620 21:49:04.100507 18600 heartbeater.cc:644] Failed to heartbeat to 127.1.155.190:36535 (0 consecutive failures): Network error: Failed to ping master at 127.1.155.190:36535: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
I20260620 21:49:04.101063 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:04.100844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:58477)
0620 21:49:04.100964 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:04.101010 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 21:49:04.101724 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:04.101509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:47107)
0620 21:49:04.101630 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:04.101676 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 21:49:04.101974 18600 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 21:49:04.104916 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 18470
I20260620 21:49:04.777619 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:04.777154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:60735)
0620 21:49:04.777447 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:04.777508 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
I20260620 21:49:04.913964 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:04.913510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:48141)
0620 21:49:04.913751 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:04.913806 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260620 21:49:04.991772 18606 raft_consensus.cc:482] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:49:04.992029 18606 raft_consensus.cc:504] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:49:04.993173 18606 leader_election.cc:288] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845), f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431)
Jun 20 21:49:05 dist-test-slave-5bvn krb5kdc[17019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992143, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.131@KRBTEST.COM
Jun 20 21:49:05 dist-test-slave-5bvn krb5kdc[17019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992143, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.130@KRBTEST.COM
I20260620 21:49:05.009616 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:04.993420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:43431 (local address 127.1.155.129:59237)
0620 21:49:04.993775 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:04.993789 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:04.993915 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:04.995740 (+ 1825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:04.995746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:04.996378 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:04.997219 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:04.997241 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:04.999171 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:04.999174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:05.000799 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:05.000810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:05.001763 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:05.001766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:05.002085 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:05.002969 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:05.002992 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:05.004764 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:05.007105 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:05.007113 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:05.007127 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:05.007530 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:05.007884 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:05.007887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:05.007890 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:05.008029 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:05.008645 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:05.008651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:05.008899 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 21:49:05.009080 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 21:49:05.009407 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":390,"spinlock_wait_cycles":18304,"thread_start_us":70,"threads_started":1}
I20260620 21:49:05.009649 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:04.993607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:41725 (local address 127.1.155.130:43845)
0620 21:49:04.993961 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:04.993988 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:04.994032 (+ 44us) server_negotiation.cc:407] Connection header received
0620 21:49:04.994886 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:04.994892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:04.995313 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:04.995637 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:04.997412 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:04.999007 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:05.000924 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:05.001635 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:05.001797 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:05.005358 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:05.005362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:05.005372 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:05.005399 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:05.007028 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:05.007690 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:05.007695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:05.007698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:05.007805 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:05.008152 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:05.008155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:05.008156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:05.008521 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:05.008660 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:05.009197 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 21:49:05.009463 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":112,"threads_started":1}
I20260620 21:49:05.009691 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:04.993576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:59237 (local address 127.1.155.131:43431)
0620 21:49:04.994035 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:04.994044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:04.994074 (+ 30us) server_negotiation.cc:407] Connection header received
0620 21:49:04.994870 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:04.994880 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:04.995313 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:04.995637 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:04.997412 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:04.999007 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:05.000924 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:05.001632 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:05.001797 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:05.004988 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:05.004995 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:05.005011 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:05.005044 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:05.006985 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:05.007646 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:05.007655 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:05.007660 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:05.007788 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:05.008133 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:05.008137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:05.008139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:05.008523 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:05.008666 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:05.009225 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 21:49:05.009496 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":135,"threads_started":1}
I20260620 21:49:05.009707 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:04.993352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:43845 (local address 127.1.155.129:41725)
0620 21:49:04.993537 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:04.993554 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:04.993669 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:04.995740 (+ 2071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:04.995746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:04.996378 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:04.997219 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:04.997241 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:04.999173 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:04.999176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:05.000799 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:05.000810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:05.001763 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:05.001766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:05.002094 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:05.003009 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:05.003020 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:05.005187 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:05.007180 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:05.007183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:05.007184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:05.007541 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:05.007937 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:05.007940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:05.007941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:05.008034 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:05.008648 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:05.008657 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:05.008900 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 21:49:05.009081 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 21:49:05.009496 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":648,"spinlock_wait_cycles":53632}
W20260620 21:49:05.012612 18221 leader_election.cc:341] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845): Not found: Tablet not found: 893c4abc4aa745b0a8218aaff4d0963c
W20260620 21:49:05.013350 18220 leader_election.cc:341] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431): Not found: Tablet not found: 893c4abc4aa745b0a8218aaff4d0963c
I20260620 21:49:05.013437 18220 leader_election.cc:302] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ab36752164b41e282202c4762838338; no voters: 49f3320599cc4370b7ec5e84a692450d, f29d3ae3dfc14c5282ecfc045bd73991
I20260620 21:49:05.013725 18606 raft_consensus.cc:2726] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:49:05.102911 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:05.102522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:49509)
0620 21:49:05.102772 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:05.102824 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
I20260620 21:49:05.778918 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:05.778478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:50505)
0620 21:49:05.778735 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:05.778819 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
I20260620 21:49:05.915066 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:05.914612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:55741)
0620 21:49:05.914922 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:05.914976 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":100,"threads_started":1}
I20260620 21:49:06.104000 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:06.103599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:43949)
0620 21:49:06.103839 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:06.103900 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":67,"threads_started":1}
I20260620 21:49:06.780047 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:06.779589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:45647)
0620 21:49:06.779863 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:06.779953 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":72,"threads_started":1}
I20260620 21:49:06.862807 18615 raft_consensus.cc:482] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:49:06.862942 18615 raft_consensus.cc:504] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:49:06.863380 18615 leader_election.cc:288] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845), f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431)
W20260620 21:49:06.864178 18221 leader_election.cc:341] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845): Not found: Tablet not found: 893c4abc4aa745b0a8218aaff4d0963c
W20260620 21:49:06.864269 18220 leader_election.cc:341] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431): Not found: Tablet not found: 893c4abc4aa745b0a8218aaff4d0963c
I20260620 21:49:06.864378 18220 leader_election.cc:302] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ab36752164b41e282202c4762838338; no voters: 49f3320599cc4370b7ec5e84a692450d, f29d3ae3dfc14c5282ecfc045bd73991
I20260620 21:49:06.864581 18615 raft_consensus.cc:2726] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:49:06.916297 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:06.915861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:54861)
0620 21:49:06.916122 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:06.916186 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
I20260620 21:49:07.105239 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:07.104829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:56811)
0620 21:49:07.105093 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:07.105151 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":72,"threads_started":1}
I20260620 21:49:07.781319 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:07.780825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:52487)
0620 21:49:07.781147 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:07.781207 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":76,"threads_started":1}
I20260620 21:49:07.917663 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:07.917132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:42759)
0620 21:49:07.917504 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:07.917568 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":100,"threads_started":1}
I20260620 21:49:08.106709 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:08.106158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:45587)
0620 21:49:08.106523 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:08.106612 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":94,"threads_started":1}
I20260620 21:49:08.783017 18621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:08.782557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:56541)
0620 21:49:08.782857 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:08.782919 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":74,"threads_started":1}
I20260620 21:49:08.919291 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:08.918663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:47687)
0620 21:49:08.919053 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:08.919153 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":85,"threads_started":1}
I20260620 21:49:09.108032 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:09.107587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:45607)
0620 21:49:09.107868 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:09.107936 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260620 21:49:09.530689 18624 raft_consensus.cc:482] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:49:09.530823 18624 raft_consensus.cc:504] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:49:09.531315 18624 leader_election.cc:288] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845), f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431)
W20260620 21:49:09.532774 18220 leader_election.cc:341] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431): Not found: Tablet not found: 893c4abc4aa745b0a8218aaff4d0963c
W20260620 21:49:09.532886 18221 leader_election.cc:341] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845): Not found: Tablet not found: 893c4abc4aa745b0a8218aaff4d0963c
I20260620 21:49:09.532994 18221 leader_election.cc:302] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ab36752164b41e282202c4762838338; no voters: 49f3320599cc4370b7ec5e84a692450d, f29d3ae3dfc14c5282ecfc045bd73991
I20260620 21:49:09.533176 18624 raft_consensus.cc:2726] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:49:09.784556 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:09.783948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:33679)
0620 21:49:09.784282 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:09.784371 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":91,"threads_started":1}
I20260620 21:49:09.921011 18626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:09.920500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:59561)
0620 21:49:09.920841 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:09.920907 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":105,"threads_started":1}
I20260620 21:49:10.109411 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:10.108940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:49657)
0620 21:49:10.109245 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:10.109308 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":81,"threads_started":1}
I20260620 21:49:10.786195 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:10.785697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:46413)
0620 21:49:10.786033 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:10.786094 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":72,"threads_started":1}
I20260620 21:49:10.922338 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:10.921799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:36879)
0620 21:49:10.922187 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:10.922245 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":83,"threads_started":1}
I20260620 21:49:11.088346 18630 raft_consensus.cc:482] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:49:11.088476 18630 raft_consensus.cc:504] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:49:11.088892 18630 leader_election.cc:288] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845), f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431)
W20260620 21:49:11.089656 18221 leader_election.cc:341] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845): Not found: Tablet not found: 893c4abc4aa745b0a8218aaff4d0963c
W20260620 21:49:11.089748 18220 leader_election.cc:341] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431): Not found: Tablet not found: 893c4abc4aa745b0a8218aaff4d0963c
I20260620 21:49:11.089879 18220 leader_election.cc:302] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ab36752164b41e282202c4762838338; no voters: 49f3320599cc4370b7ec5e84a692450d, f29d3ae3dfc14c5282ecfc045bd73991
I20260620 21:49:11.090039 18630 raft_consensus.cc:2726] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:49:11.110723 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:11.110283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:53285)
0620 21:49:11.110565 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:11.110624 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":71,"threads_started":1}
I20260620 21:49:11.787446 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:11.787000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:35779)
0620 21:49:11.787255 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:11.787309 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260620 21:49:11.923668 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:11.923205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:49617)
0620 21:49:11.923488 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:11.923558 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":69,"threads_started":1}
I20260620 21:49:12.112164 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:12.111630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:35691)
0620 21:49:12.112014 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:12.112073 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":212,"threads_started":1}
I20260620 21:49:12.788632 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:12.788238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:35979)
0620 21:49:12.788481 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:12.788537 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260620 21:49:12.925158 18636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:12.924729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:36891)
0620 21:49:12.924995 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:12.925064 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
I20260620 21:49:13.113476 18637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:13.113019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:44135)
0620 21:49:13.113294 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:13.113378 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":107,"threads_started":1}
I20260620 21:49:13.790086 18638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:13.789559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:42355)
0620 21:49:13.789921 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:13.789989 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":78,"threads_started":1}
I20260620 21:49:13.926512 18639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:13.926062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:47633)
0620 21:49:13.926357 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:13.926419 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":119,"threads_started":1}
I20260620 21:49:14.114841 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:14.114390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:59587)
0620 21:49:14.114675 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:14.114741 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":87,"threads_started":1}
I20260620 21:49:14.791359 18641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:14.790893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:41579)
0620 21:49:14.791204 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:14.791262 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":82,"threads_started":1}
I20260620 21:49:14.927790 18642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:14.927303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:47997)
0620 21:49:14.927601 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:14.927679 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":76,"threads_started":1}
I20260620 21:49:15.116106 18643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:15.115609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:38493)
0620 21:49:15.115930 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:15.115998 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":75,"threads_started":1}
I20260620 21:49:15.274784 18644 raft_consensus.cc:482] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:49:15.274914 18644 raft_consensus.cc:504] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f3320599cc4370b7ec5e84a692450d" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43845 } } peers { permanent_uuid: "f29d3ae3dfc14c5282ecfc045bd73991" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 43431 } } peers { permanent_uuid: "3ab36752164b41e282202c4762838338" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 36685 } }
I20260620 21:49:15.275404 18644 leader_election.cc:288] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845), f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431)
W20260620 21:49:15.276296 18220 leader_election.cc:341] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f29d3ae3dfc14c5282ecfc045bd73991 (127.1.155.131:43431): Not found: Tablet not found: 893c4abc4aa745b0a8218aaff4d0963c
W20260620 21:49:15.276398 18221 leader_election.cc:341] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49f3320599cc4370b7ec5e84a692450d (127.1.155.130:43845): Not found: Tablet not found: 893c4abc4aa745b0a8218aaff4d0963c
I20260620 21:49:15.276459 18221 leader_election.cc:302] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ab36752164b41e282202c4762838338; no voters: 49f3320599cc4370b7ec5e84a692450d, f29d3ae3dfc14c5282ecfc045bd73991
I20260620 21:49:15.276623 18644 raft_consensus.cc:2726] T 893c4abc4aa745b0a8218aaff4d0963c P 3ab36752164b41e282202c4762838338 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:49:15.792738 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:15.792261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:47667)
0620 21:49:15.792555 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:15.792637 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
I20260620 21:49:15.929196 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:15.928667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:59403)
0620 21:49:15.928982 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:15.929068 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":114,"threads_started":1}
I20260620 21:49:16.117413 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:16.116925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:36715)
0620 21:49:16.117181 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:16.117281 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260620 21:49:16.794261 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:16.793530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:34431)
0620 21:49:16.793818 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:16.794159 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":80,"threads_started":1}
I20260620 21:49:16.930534 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:16.930094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:33927)
0620 21:49:16.930371 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:16.930437 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260620 21:49:17.118934 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:17.118470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:37273)
0620 21:49:17.118757 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:17.118841 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":78,"threads_started":1}
I20260620 21:49:17.795742 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:17.795250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:36795)
0620 21:49:17.795577 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:17.795642 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":75,"threads_started":1}
I20260620 21:49:17.931908 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:17.931419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:45135)
0620 21:49:17.931741 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:17.931811 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":75,"threads_started":1}
I20260620 21:49:18.120358 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:18.119899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:38711)
0620 21:49:18.120201 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:18.120264 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":108,"threads_started":1}
I20260620 21:49:18.797417 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:18.796922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.129:40669)
0620 21:49:18.797245 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:18.797317 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":74,"threads_started":1}
I20260620 21:49:18.933485 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:18.932994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.130:34187)
0620 21:49:18.933317 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:18.933377 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":122,"threads_started":1}
I20260620 21:49:19.114619 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 18164
I20260620 21:49:19.120590 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 18338
I20260620 21:49:19.123118 18656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:19.122601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36535 (local address 127.1.155.131:41285)
0620 21:49:19.122922 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:19.122983 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.190:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":65,"threads_started":1}
I20260620 21:49:19.127996 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 18470
2026-06-20T21:49:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59339 ms)
[----------] 30 tests from SecurityITest (239053 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:19 dist-test-slave-5bvn krb5kdc[18659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:19 dist-test-slave-5bvn krb5kdc[18659](info): set up 2 sockets
Jun 20 21:49:19 dist-test-slave-5bvn krb5kdc[18659](info): commencing operation
krb5kdc: starting...
W20260620 21:49:20.193756 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:20 dist-test-slave-5bvn krb5kdc[18659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:49:20.322714 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:44135
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:46385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:44135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:20.434505 18672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:20.434772 18672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:20.434813 18672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:20.439133 18672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:20.439200 18672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:20.439220 18672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:20.439237 18672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:20.439253 18672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:20.444661 18672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:46385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:44135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:44135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:20.445780 18672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:20.446825 18672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:20.455296 18678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:20.455291 18677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:20.455338 18672 server_base.cc:1034] running on GCE node
W20260620 21:49:20.455291 18680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:20.456110 18672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:20.457221 18672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:20.458469 18672 hybrid_clock.cc:648] HybridClock initialized: now 1781992160458433 us; error 55 us; skew 500 ppm
Jun 20 21:49:20 dist-test-slave-5bvn krb5kdc[18659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992160, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:20.461215 18672 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:49:20.462299 18672 webserver.cc:460] Webserver started at http://127.1.155.190:38723/ using document root <none> and password file <none>
I20260620 21:49:20.462821 18672 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:20.462877 18672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:20.463037 18672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:20.465039 18672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "238eb4cae2f1433dbb4f71ca7af9af97"
format_stamp: "Formatted at 2026-06-20 21:49:20 on dist-test-slave-5bvn"
server_key: "8a1b2c19d6cbd55aa9c7c731e3e71d4d"
server_key_iv: "d83ca092bbd8b4dc85d9a95cf0fe48f9"
server_key_version: "encryptionkey@0"
I20260620 21:49:20.465540 18672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "238eb4cae2f1433dbb4f71ca7af9af97"
format_stamp: "Formatted at 2026-06-20 21:49:20 on dist-test-slave-5bvn"
server_key: "8a1b2c19d6cbd55aa9c7c731e3e71d4d"
server_key_iv: "d83ca092bbd8b4dc85d9a95cf0fe48f9"
server_key_version: "encryptionkey@0"
I20260620 21:49:20.469317 18672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:49:20.472062 18687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:20.473464 18672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:49:20.473575 18672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "238eb4cae2f1433dbb4f71ca7af9af97"
format_stamp: "Formatted at 2026-06-20 21:49:20 on dist-test-slave-5bvn"
server_key: "8a1b2c19d6cbd55aa9c7c731e3e71d4d"
server_key_iv: "d83ca092bbd8b4dc85d9a95cf0fe48f9"
server_key_version: "encryptionkey@0"
I20260620 21:49:20.473691 18672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:20.488621 18672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:20.492045 18672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:20.492216 18672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:20.499574 18672 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:44135
I20260620 21:49:20.499604 18739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:44135 every 8 connection(s)
I20260620 21:49:20.500519 18672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:20.503291 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:20.508693 18740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97: Bootstrap starting.
I20260620 21:49:20.509279 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 18672
I20260620 21:49:20.509440 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:20.509742 1646 external_mini_cluster.cc:1428] Setting key a0310633fce1ff7083eded1bc9cd3767
I20260620 21:49:20.511554 18740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:20.512285 18740 log.cc:824] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:20.514487 18740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97: No bootstrap required, opened a new log
I20260620 21:49:20.518965 18740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238eb4cae2f1433dbb4f71ca7af9af97" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 44135 } }
Jun 20 21:49:20 dist-test-slave-5bvn krb5kdc[18659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:20.519212 18740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:20.519304 18740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 238eb4cae2f1433dbb4f71ca7af9af97, State: Initialized, Role: FOLLOWER
I20260620 21:49:20.519820 18740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238eb4cae2f1433dbb4f71ca7af9af97" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 44135 } }
I20260620 21:49:20.519950 18740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:20.520025 18740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:20.520120 18740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:20.521013 18740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238eb4cae2f1433dbb4f71ca7af9af97" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 44135 } }
I20260620 21:49:20.521358 18740 leader_election.cc:302] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 238eb4cae2f1433dbb4f71ca7af9af97; no voters:
I20260620 21:49:20.521673 18740 leader_election.cc:288] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:20.521798 18745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:20.522032 18745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [term 1 LEADER]: Becoming Leader. State: Replica: 238eb4cae2f1433dbb4f71ca7af9af97, State: Running, Role: LEADER
I20260620 21:49:20.522404 18745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238eb4cae2f1433dbb4f71ca7af9af97" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 44135 } }
I20260620 21:49:20.523094 18740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:20.524137 18746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "238eb4cae2f1433dbb4f71ca7af9af97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238eb4cae2f1433dbb4f71ca7af9af97" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 44135 } } }
I20260620 21:49:20.524286 18746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:20.524681 18747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 238eb4cae2f1433dbb4f71ca7af9af97. Latest consensus state: current_term: 1 leader_uuid: "238eb4cae2f1433dbb4f71ca7af9af97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238eb4cae2f1433dbb4f71ca7af9af97" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 44135 } } }
I20260620 21:49:20.524840 18747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:20.525454 18754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:20.526188 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:20.511029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35900 (local address 127.1.155.190:44135)
0620 21:49:20.511478 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:20.511492 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:20.511535 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:49:20.512261 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:20.512278 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:20.512698 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:20.513079 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:20.513991 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:20.515085 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:20.516282 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:20.516772 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:20.516899 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:20.519821 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:20.519827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:20.519839 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:20.519865 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:20.522133 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:20.522630 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:20.522636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:20.522642 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:20.522777 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:20.523108 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:20.523112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:20.523113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:20.523620 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:20.523834 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:20.524802 (+ 968us) server_negotiation.cc:299] Negotiation successful
0620 21:49:20.525101 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":131,"threads_started":1}
I20260620 21:49:20.528348 18754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:20.534029 18754 catalog_manager.cc:1347] Generated new cluster ID: 0c1b7b7f7b144a798146ccea95b8678b
I20260620 21:49:20.534111 18754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:20.550138 18754 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:20.551463 18754 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:20.560133 18754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97: Generated new TSK 0
I20260620 21:49:20.560904 18754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:49:20.597780 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:44135
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:46385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:20.716351 18766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:20.716607 18766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:20.716650 18766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:20.721026 18766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:20.721113 18766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:20.721174 18766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:49:20.727725 18766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:46385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:44135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:20.729588 18766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:20.730785 18766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:20.741123 18771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:20.741125 18772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:20.741178 18766 server_base.cc:1034] running on GCE node
W20260620 21:49:20.741123 18774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:20.741909 18766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:20.742571 18766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:20.743815 18766 hybrid_clock.cc:648] HybridClock initialized: now 1781992160743799 us; error 26 us; skew 500 ppm
Jun 20 21:49:20 dist-test-slave-5bvn krb5kdc[18659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992160, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:20.747121 18766 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:49:20.748047 18766 webserver.cc:460] Webserver started at http://127.1.155.129:34231/ using document root <none> and password file <none>
I20260620 21:49:20.748624 18766 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:20.748723 18766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:20.748953 18766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:20.751032 18766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4"
format_stamp: "Formatted at 2026-06-20 21:49:20 on dist-test-slave-5bvn"
server_key: "1ddb3d3b59c71dabf78a554683884fc7"
server_key_iv: "f648e44810af37bd4944e9e589dc817a"
server_key_version: "encryptionkey@0"
I20260620 21:49:20.751621 18766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4"
format_stamp: "Formatted at 2026-06-20 21:49:20 on dist-test-slave-5bvn"
server_key: "1ddb3d3b59c71dabf78a554683884fc7"
server_key_iv: "f648e44810af37bd4944e9e589dc817a"
server_key_version: "encryptionkey@0"
I20260620 21:49:20.755532 18766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 21:49:20.758090 18781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:20.759315 18766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:49:20.759500 18766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4"
format_stamp: "Formatted at 2026-06-20 21:49:20 on dist-test-slave-5bvn"
server_key: "1ddb3d3b59c71dabf78a554683884fc7"
server_key_iv: "f648e44810af37bd4944e9e589dc817a"
server_key_version: "encryptionkey@0"
I20260620 21:49:20.759624 18766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:20.773463 18766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:20.777123 18766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:20.777349 18766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:20.778107 18766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:20.779104 18766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:20.779163 18766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:20.779211 18766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:20.779230 18766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:20.789149 18766 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:41339
I20260620 21:49:20.789166 18894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:41339 every 8 connection(s)
I20260620 21:49:20.790134 18766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:49:20.795128 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 18766
I20260620 21:49:20.795310 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:49:20.795614 1646 external_mini_cluster.cc:1428] Setting key 37f1171173ed3781dda07f6ca9a265ed
Jun 20 21:49:20 dist-test-slave-5bvn krb5kdc[18659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992160, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
I20260620 21:49:20.804684 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:20.792054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:49343 (local address 127.1.155.190:44135)
0620 21:49:20.792235 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:20.792240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:20.793053 (+ 813us) server_negotiation.cc:407] Connection header received
0620 21:49:20.794329 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:20.794332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:20.794380 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:20.794459 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:20.796000 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:20.796524 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:20.797744 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:20.798050 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:20.798167 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:20.800898 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:20.800902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:20.800904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:20.800930 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:20.802561 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:20.803112 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:20.803117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:20.803118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:20.803200 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:20.803548 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:20.803553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:20.803554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:20.803738 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:20.803851 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:20.804376 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 21:49:20.804514 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 21:49:20.805545 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:20.792444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:44135 (local address 127.1.155.129:49343)
0620 21:49:20.792894 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:20.792936 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:20.794110 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:20.794625 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:20.794632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:20.795134 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:20.795824 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:20.795845 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:20.796692 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:20.796695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:20.797605 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:20.797615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:20.798204 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:20.798208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:20.798384 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:20.798999 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:20.799015 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:20.800706 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:20.802697 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:20.802704 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:20.802720 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:20.802997 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:20.803298 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:20.803302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:20.803304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:20.803447 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:20.803850 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:20.803859 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:20.804134 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 21:49:20.804333 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 21:49:20.804663 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":99,"threads_started":1}
I20260620 21:49:20.807061 18895 heartbeater.cc:347] Connected to a master server at 127.1.155.190:44135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
I20260620 21:49:20.807401 18895 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:20.807898 18895 heartbeater.cc:511] Master 127.1.155.190:44135 requested a full tablet report, sending...
I20260620 21:49:20.809777 18704 ts_manager.cc:194] Registered new tserver with Master: 2f2b3aaa4a7544c9aeaa5497c0251cd4 (127.1.155.129:41339)
I20260620 21:49:20.811728 18704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:49343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:49:20.827029 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:44135
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:46385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:20.938030 18900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:20.938283 18900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:20.938374 18900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:20.942577 18900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:20.942636 18900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:20.942741 18900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:49:20.948401 18900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:46385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:44135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:20.949538 18900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:20.950656 18900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:20.960269 18906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:20.960275 18908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:20.960290 18905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:20.960613 18900 server_base.cc:1034] running on GCE node
I20260620 21:49:20.961063 18900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:20.961755 18900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:20.962940 18900 hybrid_clock.cc:648] HybridClock initialized: now 1781992160962928 us; error 24 us; skew 500 ppm
Jun 20 21:49:20 dist-test-slave-5bvn krb5kdc[18659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992160, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:20.965662 18900 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:49:20.966681 18900 webserver.cc:460] Webserver started at http://127.1.155.130:35265/ using document root <none> and password file <none>
I20260620 21:49:20.967233 18900 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:20.967326 18900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:20.967549 18900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:20.969491 18900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "8b9798ab91f249f1acdeb056801d1045"
format_stamp: "Formatted at 2026-06-20 21:49:20 on dist-test-slave-5bvn"
server_key: "08e2cfc240df494a2eb33663ea350678"
server_key_iv: "6804836f1f40ab80c7918d516ef8688c"
server_key_version: "encryptionkey@0"
I20260620 21:49:20.970072 18900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "8b9798ab91f249f1acdeb056801d1045"
format_stamp: "Formatted at 2026-06-20 21:49:20 on dist-test-slave-5bvn"
server_key: "08e2cfc240df494a2eb33663ea350678"
server_key_iv: "6804836f1f40ab80c7918d516ef8688c"
server_key_version: "encryptionkey@0"
I20260620 21:49:20.973603 18900 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:49:20.975963 18915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:20.977295 18900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:49:20.977419 18900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "8b9798ab91f249f1acdeb056801d1045"
format_stamp: "Formatted at 2026-06-20 21:49:20 on dist-test-slave-5bvn"
server_key: "08e2cfc240df494a2eb33663ea350678"
server_key_iv: "6804836f1f40ab80c7918d516ef8688c"
server_key_version: "encryptionkey@0"
I20260620 21:49:20.977527 18900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:20.991593 18900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:20.994562 18900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:20.994765 18900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:20.995343 18900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:20.996133 18900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:20.996215 18900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:20.996280 18900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:20.996318 18900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:21.005103 18900 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:41593
I20260620 21:49:21.005134 19028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:41593 every 8 connection(s)
I20260620 21:49:21.006011 18900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:49:21.013630 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 18900
I20260620 21:49:21.013767 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:49:21.014123 1646 external_mini_cluster.cc:1428] Setting key 22c8e5e86af5636004991c49c01f2c52
Jun 20 21:49:21 dist-test-slave-5bvn krb5kdc[18659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992160, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:21.020712 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.007810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:48463 (local address 127.1.155.190:44135)
0620 21:49:21.007964 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:21.007969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:21.008794 (+ 825us) server_negotiation.cc:407] Connection header received
0620 21:49:21.009977 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:21.009979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:21.010025 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:21.010098 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:21.011105 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.011635 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.012738 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.012961 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.013062 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.015949 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:21.015958 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:21.015961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:21.016204 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:21.018132 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:21.018867 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:21.018870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:21.018871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:21.018960 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:21.019422 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:21.019424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:21.019426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:21.019627 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:21.019708 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:21.020475 (+ 767us) server_negotiation.cc:299] Negotiation successful
0620 21:49:21.020586 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:49:21.022111 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.008223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:44135 (local address 127.1.155.130:48463)
0620 21:49:21.008641 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:21.008682 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:21.009752 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:21.010217 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:21.010223 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:21.010537 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:21.010983 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:21.010994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.011751 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.011754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:21.012601 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:21.012609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.013100 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.013103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:21.013236 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.013933 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:21.013950 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:21.015757 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:21.018282 (+ 2525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:21.018289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:21.018306 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:21.018724 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:21.019073 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:21.019079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:21.019084 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:21.019285 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:21.019752 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:21.019762 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:21.020143 (+ 381us) client_negotiation.cc:769] Sending connection context
0620 21:49:21.020395 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 21:49:21.020791 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
I20260620 21:49:21.023451 19029 heartbeater.cc:347] Connected to a master server at 127.1.155.190:44135
I20260620 21:49:21.023748 19029 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:21.024413 19029 heartbeater.cc:511] Master 127.1.155.190:44135 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
I20260620 21:49:21.025609 18704 ts_manager.cc:194] Registered new tserver with Master: 8b9798ab91f249f1acdeb056801d1045 (127.1.155.130:41593)
I20260620 21:49:21.026260 18704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:48463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:49:21.046273 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:44135
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:46385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:21.159067 19034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:21.159377 19034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:21.159462 19034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:21.163681 19034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:21.163775 19034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:21.163861 19034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:49:21.169433 19034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:46385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:44135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:21.170589 19034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:21.171625 19034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:21.181231 19039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:21.181260 19040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:21.181442 19034 server_base.cc:1034] running on GCE node
W20260620 21:49:21.181226 19042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:21.182029 19034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:21.182642 19034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:21.183838 19034 hybrid_clock.cc:648] HybridClock initialized: now 1781992161183822 us; error 28 us; skew 500 ppm
Jun 20 21:49:21 dist-test-slave-5bvn krb5kdc[18659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992161, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:21.187088 19034 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:49:21.188066 19034 webserver.cc:460] Webserver started at http://127.1.155.131:37441/ using document root <none> and password file <none>
I20260620 21:49:21.188661 19034 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:21.188723 19034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:21.188953 19034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:21.191100 19034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "b4be7037e0834b58a29fda17a0c77580"
format_stamp: "Formatted at 2026-06-20 21:49:21 on dist-test-slave-5bvn"
server_key: "29b7635df1df7ee59824223cbfab398b"
server_key_iv: "8453b725c6e02e684c7bbdc07758d8e9"
server_key_version: "encryptionkey@0"
I20260620 21:49:21.191687 19034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "b4be7037e0834b58a29fda17a0c77580"
format_stamp: "Formatted at 2026-06-20 21:49:21 on dist-test-slave-5bvn"
server_key: "29b7635df1df7ee59824223cbfab398b"
server_key_iv: "8453b725c6e02e684c7bbdc07758d8e9"
server_key_version: "encryptionkey@0"
I20260620 21:49:21.195451 19034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 21:49:21.198076 19049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:21.199373 19034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:49:21.199517 19034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "b4be7037e0834b58a29fda17a0c77580"
format_stamp: "Formatted at 2026-06-20 21:49:21 on dist-test-slave-5bvn"
server_key: "29b7635df1df7ee59824223cbfab398b"
server_key_iv: "8453b725c6e02e684c7bbdc07758d8e9"
server_key_version: "encryptionkey@0"
I20260620 21:49:21.199632 19034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:21.226011 19034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:21.229018 19034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:21.229254 19034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:21.229951 19034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:21.230772 19034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:21.230819 19034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:21.230899 19034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:21.230942 19034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:21.240239 19034 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:39067
I20260620 21:49:21.240269 19162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:39067 every 8 connection(s)
I20260620 21:49:21.241155 19034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:49:21.243268 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 19034
I20260620 21:49:21.243419 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:49:21.243732 1646 external_mini_cluster.cc:1428] Setting key 039d4977dbf554cfb20e0816958113a1
Jun 20 21:49:21 dist-test-slave-5bvn krb5kdc[18659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992161, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:21.256147 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.243146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:42005 (local address 127.1.155.190:44135)
0620 21:49:21.243328 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:21.243334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:21.244269 (+ 935us) server_negotiation.cc:407] Connection header received
0620 21:49:21.245852 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:21.245856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:21.245913 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:21.245989 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:21.247146 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.247679 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.248818 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.249047 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.249165 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.251867 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:21.251871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:21.251874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:21.251903 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:21.253732 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:21.254357 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:21.254362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:21.254364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:21.254450 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:21.254857 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:21.254860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:21.254862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:21.255054 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:21.255169 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:21.255828 (+ 659us) server_negotiation.cc:299] Negotiation successful
0620 21:49:21.255960 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 21:49:21.257319 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.243572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:44135 (local address 127.1.155.131:42005)
0620 21:49:21.244116 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:21.244159 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:21.245611 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:21.246156 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:21.246162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:21.246524 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:21.247003 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:21.247015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.247812 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.247815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:21.248690 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:21.248698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.249203 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.249208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:21.249382 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.250066 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:21.250084 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:21.251629 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:21.253873 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:21.253882 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:21.253900 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:21.254264 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:21.254554 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:21.254560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:21.254563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:21.254754 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:21.255167 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:21.255174 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:21.255522 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 21:49:21.255773 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 21:49:21.256108 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":128,"threads_started":1}
I20260620 21:49:21.258757 19163 heartbeater.cc:347] Connected to a master server at 127.1.155.190:44135
I20260620 21:49:21.259029 19163 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:21.259703 19163 heartbeater.cc:511] Master 127.1.155.190:44135 requested a full tablet report, sending...
I20260620 21:49:21.260953 18704 ts_manager.cc:194] Registered new tserver with Master: b4be7037e0834b58a29fda17a0c77580 (127.1.155.131:39067)
I20260620 21:49:21.261577 18704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:42005
I20260620 21:49:21.272006 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:49:21.282128 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.274195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35908 (local address 127.1.155.190:44135)
0620 21:49:21.274351 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:21.274356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:21.274495 (+ 139us) server_negotiation.cc:407] Connection header received
0620 21:49:21.274598 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:21.274600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:21.274649 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:21.274716 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:21.275473 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.275982 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.277012 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.277256 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.277367 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.278810 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:21.278813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:21.278815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:21.278845 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:21.280315 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:21.280852 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:21.280856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:21.280858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:21.280940 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:21.281221 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:21.281226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:21.281227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:21.281450 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:21.281592 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:21.281828 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 21:49:21.282007 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 21:49:21.284950 18704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:49:21.288091 18704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:49:21.303145 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.296789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:41593 (local address 127.0.0.1:60324)
0620 21:49:21.297647 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:21.297701 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:21.297877 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:21.298762 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:21.298768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:21.298790 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:21.298938 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:21.298945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.300295 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.300299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:21.301437 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:21.301448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.302571 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.302582 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:21.302664 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.302695 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 21:49:21.302859 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 21:49:21.302985 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260620 21:49:21.303195 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.296928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.1.155.130:41593)
0620 21:49:21.298204 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:21.298220 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:21.298244 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:49:21.298338 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:21.298344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:21.298581 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:21.298759 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:21.299068 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.299908 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.301597 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.302426 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.302552 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.302707 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:21.302884 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 21:49:21.302984 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1167,"thread_start_us":110,"threads_started":1}
I20260620 21:49:21.304554 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.296827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:39067 (local address 127.0.0.1:38254)
0620 21:49:21.298148 (+ 1321us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:21.298163 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:21.298249 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:21.298928 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:21.298930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:21.298945 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:21.299047 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:21.299052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.300520 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.300523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:21.301964 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:21.301975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.304087 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.304090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:21.304187 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.304218 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 21:49:21.304309 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 21:49:21.304392 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1254,"mutex_wait_us":162,"thread_start_us":123,"threads_started":1}
I20260620 21:49:21.305038 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.297588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.1.155.131:39067)
0620 21:49:21.298349 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:21.298374 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:21.298396 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:49:21.298471 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:21.298481 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:21.298621 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:21.298781 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:21.299165 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.300351 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.302794 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.303727 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.303850 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.303994 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:21.304762 (+ 768us) server_negotiation.cc:299] Negotiation successful
0620 21:49:21.304875 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":175,"threads_started":1}
I20260620 21:49:21.305145 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.297092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:41339 (local address 127.0.0.1:51152)
0620 21:49:21.298174 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:21.298189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:21.298346 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:21.298966 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:21.298970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:21.299018 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:21.299127 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:21.299134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.300527 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.300529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:21.301964 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:21.301975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.304480 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.304485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:21.304594 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.304838 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 21:49:21.304921 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 21:49:21.304995 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"mutex_wait_us":77,"thread_start_us":87,"threads_started":1}
I20260620 21:49:21.305513 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.297721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51152 (local address 127.1.155.129:41339)
0620 21:49:21.298155 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:21.298173 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:21.298274 (+ 101us) server_negotiation.cc:407] Connection header received
0620 21:49:21.298509 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:21.298515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:21.298671 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:21.298830 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:21.299270 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.300395 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.302098 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.302944 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.303058 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.303189 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:21.305252 (+ 2063us) server_negotiation.cc:299] Negotiation successful
0620 21:49:21.305383 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":125,"threads_started":1}
I20260620 21:49:21.306665 18963 tablet_service.cc:1467] Processing CreateTablet for tablet a33327c2d0fb44d9b3287fb08d2479de (DEFAULT_TABLE table=test-table [id=b25a3a9f1b7f4155b64ba6211a7fe6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:21.307832 18963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a33327c2d0fb44d9b3287fb08d2479de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:21.307804 19097 tablet_service.cc:1467] Processing CreateTablet for tablet a33327c2d0fb44d9b3287fb08d2479de (DEFAULT_TABLE table=test-table [id=b25a3a9f1b7f4155b64ba6211a7fe6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:21.308040 18829 tablet_service.cc:1467] Processing CreateTablet for tablet a33327c2d0fb44d9b3287fb08d2479de (DEFAULT_TABLE table=test-table [id=b25a3a9f1b7f4155b64ba6211a7fe6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:21.308799 19097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a33327c2d0fb44d9b3287fb08d2479de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:21.309033 18829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a33327c2d0fb44d9b3287fb08d2479de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:21.314990 19180 tablet_bootstrap.cc:492] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4: Bootstrap starting.
I20260620 21:49:21.316187 19181 tablet_bootstrap.cc:492] T a33327c2d0fb44d9b3287fb08d2479de P 8b9798ab91f249f1acdeb056801d1045: Bootstrap starting.
I20260620 21:49:21.316360 19182 tablet_bootstrap.cc:492] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580: Bootstrap starting.
I20260620 21:49:21.317497 19180 tablet_bootstrap.cc:654] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:21.318466 19182 tablet_bootstrap.cc:654] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:21.318435 19180 log.cc:824] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:21.318919 19181 tablet_bootstrap.cc:654] T a33327c2d0fb44d9b3287fb08d2479de P 8b9798ab91f249f1acdeb056801d1045: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:21.319687 19182 log.cc:824] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:21.319687 19181 log.cc:824] T a33327c2d0fb44d9b3287fb08d2479de P 8b9798ab91f249f1acdeb056801d1045: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:21.320709 19180 tablet_bootstrap.cc:492] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4: No bootstrap required, opened a new log
I20260620 21:49:21.320927 19180 ts_tablet_manager.cc:1397] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:49:21.321693 19181 tablet_bootstrap.cc:492] T a33327c2d0fb44d9b3287fb08d2479de P 8b9798ab91f249f1acdeb056801d1045: No bootstrap required, opened a new log
I20260620 21:49:21.321915 19181 ts_tablet_manager.cc:1397] T a33327c2d0fb44d9b3287fb08d2479de P 8b9798ab91f249f1acdeb056801d1045: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:49:21.322027 19182 tablet_bootstrap.cc:492] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580: No bootstrap required, opened a new log
I20260620 21:49:21.322202 19182 ts_tablet_manager.cc:1397] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:49:21.324752 19180 raft_consensus.cc:348] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9798ab91f249f1acdeb056801d1045" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41593 } } peers { permanent_uuid: "b4be7037e0834b58a29fda17a0c77580" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39067 } } peers { permanent_uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41339 } }
I20260620 21:49:21.325007 19180 raft_consensus.cc:374] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:21.325059 19180 raft_consensus.cc:729] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f2b3aaa4a7544c9aeaa5497c0251cd4, State: Initialized, Role: FOLLOWER
I20260620 21:49:21.325989 19181 raft_consensus.cc:348] T a33327c2d0fb44d9b3287fb08d2479de P 8b9798ab91f249f1acdeb056801d1045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9798ab91f249f1acdeb056801d1045" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41593 } } peers { permanent_uuid: "b4be7037e0834b58a29fda17a0c77580" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39067 } } peers { permanent_uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41339 } }
I20260620 21:49:21.325989 19182 raft_consensus.cc:348] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9798ab91f249f1acdeb056801d1045" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41593 } } peers { permanent_uuid: "b4be7037e0834b58a29fda17a0c77580" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39067 } } peers { permanent_uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41339 } }
I20260620 21:49:21.326123 19180 consensus_queue.cc:260] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9798ab91f249f1acdeb056801d1045" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41593 } } peers { permanent_uuid: "b4be7037e0834b58a29fda17a0c77580" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39067 } } peers { permanent_uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41339 } }
I20260620 21:49:21.326251 19182 raft_consensus.cc:374] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:21.326251 19181 raft_consensus.cc:374] T a33327c2d0fb44d9b3287fb08d2479de P 8b9798ab91f249f1acdeb056801d1045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:21.326306 19182 raft_consensus.cc:729] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4be7037e0834b58a29fda17a0c77580, State: Initialized, Role: FOLLOWER
I20260620 21:49:21.326306 19181 raft_consensus.cc:729] T a33327c2d0fb44d9b3287fb08d2479de P 8b9798ab91f249f1acdeb056801d1045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9798ab91f249f1acdeb056801d1045, State: Initialized, Role: FOLLOWER
I20260620 21:49:21.326732 19181 consensus_queue.cc:260] T a33327c2d0fb44d9b3287fb08d2479de P 8b9798ab91f249f1acdeb056801d1045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9798ab91f249f1acdeb056801d1045" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41593 } } peers { permanent_uuid: "b4be7037e0834b58a29fda17a0c77580" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39067 } } peers { permanent_uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41339 } }
I20260620 21:49:21.326732 19182 consensus_queue.cc:260] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9798ab91f249f1acdeb056801d1045" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41593 } } peers { permanent_uuid: "b4be7037e0834b58a29fda17a0c77580" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39067 } } peers { permanent_uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41339 } }
I20260620 21:49:21.327160 19180 ts_tablet_manager.cc:1428] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:49:21.327503 18895 heartbeater.cc:503] Master 127.1.155.190:44135 was elected leader, sending a full tablet report...
I20260620 21:49:21.327556 19029 heartbeater.cc:503] Master 127.1.155.190:44135 was elected leader, sending a full tablet report...
I20260620 21:49:21.327654 19181 ts_tablet_manager.cc:1428] T a33327c2d0fb44d9b3287fb08d2479de P 8b9798ab91f249f1acdeb056801d1045: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:49:21.328476 19163 heartbeater.cc:503] Master 127.1.155.190:44135 was elected leader, sending a full tablet report...
I20260620 21:49:21.329532 19182 ts_tablet_manager.cc:1428] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260620 21:49:21.492208 19164 tablet.cc:2369] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:49:21.494333 19187 raft_consensus.cc:482] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:49:21.494503 19187 raft_consensus.cc:504] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9798ab91f249f1acdeb056801d1045" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41593 } } peers { permanent_uuid: "b4be7037e0834b58a29fda17a0c77580" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39067 } } peers { permanent_uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41339 } }
I20260620 21:49:21.495810 19187 leader_election.cc:288] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b9798ab91f249f1acdeb056801d1045 (127.1.155.130:41593), 2f2b3aaa4a7544c9aeaa5497c0251cd4 (127.1.155.129:41339)
I20260620 21:49:21.499933 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.496053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:41593 (local address 127.1.155.131:40981)
0620 21:49:21.496256 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:21.496274 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:21.496380 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:21.496733 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:21.496736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:21.496757 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:21.496891 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:21.496898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.497752 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.497755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:21.498664 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:21.498673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.499482 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.499485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:21.499595 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.499622 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:49:21.499719 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 21:49:21.499793 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":1}
I20260620 21:49:21.499922 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.496167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:40981 (local address 127.1.155.130:41593)
0620 21:49:21.496326 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:21.496331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:21.496348 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:49:21.496528 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:21.496531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:21.496598 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:21.496683 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:21.497038 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.497624 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.498798 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.499241 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.499390 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.499449 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:21.499738 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 21:49:21.499808 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 21:49:21.499965 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.496315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:42275 (local address 127.1.155.129:41339)
0620 21:49:21.496457 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:21.496463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:21.496690 (+ 227us) server_negotiation.cc:407] Connection header received
0620 21:49:21.496751 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:21.496753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:21.496803 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:21.496897 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:21.497218 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.497770 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.498911 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.499371 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.499500 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.499551 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:21.499781 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 21:49:21.499854 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:49:21.500284 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.496134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:41339 (local address 127.1.155.131:42275)
0620 21:49:21.496535 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:21.496548 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:21.496654 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:21.496921 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:21.496924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:21.496949 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:21.497104 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:21.497110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.497927 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.497930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:21.498786 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:21.498796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.499527 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.499530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:21.499636 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.499657 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:49:21.500096 (+ 439us) client_negotiation.cc:240] Negotiation successful
0620 21:49:21.500164 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":19,"thread_start_us":142,"threads_started":1}
I20260620 21:49:21.500622 18983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33327c2d0fb44d9b3287fb08d2479de" candidate_uuid: "b4be7037e0834b58a29fda17a0c77580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9798ab91f249f1acdeb056801d1045" is_pre_election: true
I20260620 21:49:21.500967 18983 raft_consensus.cc:2454] T a33327c2d0fb44d9b3287fb08d2479de P 8b9798ab91f249f1acdeb056801d1045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4be7037e0834b58a29fda17a0c77580 in term 0.
I20260620 21:49:21.501219 18849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33327c2d0fb44d9b3287fb08d2479de" candidate_uuid: "b4be7037e0834b58a29fda17a0c77580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4" is_pre_election: true
I20260620 21:49:21.501539 19052 leader_election.cc:302] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9798ab91f249f1acdeb056801d1045, b4be7037e0834b58a29fda17a0c77580; no voters:
I20260620 21:49:21.501572 18849 raft_consensus.cc:2454] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4be7037e0834b58a29fda17a0c77580 in term 0.
I20260620 21:49:21.501969 19187 raft_consensus.cc:2781] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:49:21.502068 19187 raft_consensus.cc:482] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:49:21.502127 19187 raft_consensus.cc:3037] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:21.503203 19187 raft_consensus.cc:504] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9798ab91f249f1acdeb056801d1045" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41593 } } peers { permanent_uuid: "b4be7037e0834b58a29fda17a0c77580" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39067 } } peers { permanent_uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41339 } }
I20260620 21:49:21.503597 19187 leader_election.cc:288] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [CANDIDATE]: Term 1 election: Requested vote from peers 8b9798ab91f249f1acdeb056801d1045 (127.1.155.130:41593), 2f2b3aaa4a7544c9aeaa5497c0251cd4 (127.1.155.129:41339)
I20260620 21:49:21.503950 18849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33327c2d0fb44d9b3287fb08d2479de" candidate_uuid: "b4be7037e0834b58a29fda17a0c77580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4"
I20260620 21:49:21.503966 18983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a33327c2d0fb44d9b3287fb08d2479de" candidate_uuid: "b4be7037e0834b58a29fda17a0c77580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9798ab91f249f1acdeb056801d1045"
I20260620 21:49:21.504119 18849 raft_consensus.cc:3037] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:21.504158 18983 raft_consensus.cc:3037] T a33327c2d0fb44d9b3287fb08d2479de P 8b9798ab91f249f1acdeb056801d1045 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:21.505156 18849 raft_consensus.cc:2454] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4be7037e0834b58a29fda17a0c77580 in term 1.
I20260620 21:49:21.505434 18983 raft_consensus.cc:2454] T a33327c2d0fb44d9b3287fb08d2479de P 8b9798ab91f249f1acdeb056801d1045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4be7037e0834b58a29fda17a0c77580 in term 1.
I20260620 21:49:21.505478 19053 leader_election.cc:302] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f2b3aaa4a7544c9aeaa5497c0251cd4, b4be7037e0834b58a29fda17a0c77580; no voters:
I20260620 21:49:21.505651 19187 raft_consensus.cc:2781] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:21.505954 19187 raft_consensus.cc:686] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [term 1 LEADER]: Becoming Leader. State: Replica: b4be7037e0834b58a29fda17a0c77580, State: Running, Role: LEADER
I20260620 21:49:21.506285 19187 consensus_queue.cc:237] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9798ab91f249f1acdeb056801d1045" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41593 } } peers { permanent_uuid: "b4be7037e0834b58a29fda17a0c77580" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39067 } } peers { permanent_uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41339 } }
W20260620 21:49:21.507172 19030 tablet.cc:2369] T a33327c2d0fb44d9b3287fb08d2479de P 8b9798ab91f249f1acdeb056801d1045: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:49:21.509470 18704 catalog_manager.cc:5495] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4be7037e0834b58a29fda17a0c77580 (127.1.155.131). New cstate: current_term: 1 leader_uuid: "b4be7037e0834b58a29fda17a0c77580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9798ab91f249f1acdeb056801d1045" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4be7037e0834b58a29fda17a0c77580" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 39067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41339 } health_report { overall_health: UNKNOWN } } }
I20260620 21:49:21.539543 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.535781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.1.155.131:39067)
0620 21:49:21.535951 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:21.535956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:21.535975 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:49:21.536044 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:21.536046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:21.536096 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:21.536212 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:49:21.536499 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.536992 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.537996 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.538214 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.538335 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.538657 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:49:21.539117 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:49:21.539256 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:21.539299 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 21:49:21.539357 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 21:49:21.541637 18896 tablet.cc:2369] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:49:21.545120 18849 raft_consensus.cc:1261] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4 [term 1 FOLLOWER]: Refusing update from remote peer b4be7037e0834b58a29fda17a0c77580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:49:21.545132 18983 raft_consensus.cc:1261] T a33327c2d0fb44d9b3287fb08d2479de P 8b9798ab91f249f1acdeb056801d1045 [term 1 FOLLOWER]: Refusing update from remote peer b4be7037e0834b58a29fda17a0c77580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:49:21.545858 19187 consensus_queue.cc:1035] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f2b3aaa4a7544c9aeaa5497c0251cd4" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:49:21.546082 19190 consensus_queue.cc:1035] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9798ab91f249f1acdeb056801d1045" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:49:21.554600 19195 mvcc.cc:205] Tried to move back new op lower bound from 7299039893681631232 to 7299039893532229632. Current Snapshot: MvccSnapshot[applied={T|T < 7299039893681631232}]
I20260620 21:49:21.554754 19194 mvcc.cc:205] Tried to move back new op lower bound from 7299039893681631232 to 7299039893532229632. Current Snapshot: MvccSnapshot[applied={T|T < 7299039893681631232}]
I20260620 21:49:21.561991 18704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:49:21.562178 18704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:49:21.564782 18704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 238eb4cae2f1433dbb4f71ca7af9af97: Sending DeleteTablet for 3 replicas of tablet a33327c2d0fb44d9b3287fb08d2479de
I20260620 21:49:21.565585 18963 tablet_service.cc:1514] Processing DeleteTablet for tablet a33327c2d0fb44d9b3287fb08d2479de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:21 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:60324
I20260620 21:49:21.565580 18829 tablet_service.cc:1514] Processing DeleteTablet for tablet a33327c2d0fb44d9b3287fb08d2479de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:21 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:51152
I20260620 21:49:21.566133 19203 tablet_replica.cc:330] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4: stopping tablet replica
I20260620 21:49:21.566141 19097 tablet_service.cc:1514] Processing DeleteTablet for tablet a33327c2d0fb44d9b3287fb08d2479de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:21 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:38254
I20260620 21:49:21.566469 19203 raft_consensus.cc:2229] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:49:21.566622 19205 tablet_replica.cc:330] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580: stopping tablet replica
I20260620 21:49:21.566797 19203 raft_consensus.cc:2258] T a33327c2d0fb44d9b3287fb08d2479de P 2f2b3aaa4a7544c9aeaa5497c0251cd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:49:21.566992 19205 raft_consensus.cc:2229] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:49:21.567263 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 18766
I20260620 21:49:21.568140 19205 raft_consensus.cc:2258] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:49:21.569532 19205 ts_tablet_manager.cc:1905] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:49:21.572304 19205 ts_tablet_manager.cc:1918] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:49:21.572414 19205 log.cc:1196] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/a33327c2d0fb44d9b3287fb08d2479de
I20260620 21:49:21.572733 19205 ts_tablet_manager.cc:1939] T a33327c2d0fb44d9b3287fb08d2479de P b4be7037e0834b58a29fda17a0c77580: Deleting consensus metadata
I20260620 21:49:21.573956 18688 catalog_manager.cc:4841] TS b4be7037e0834b58a29fda17a0c77580 (127.1.155.131:39067): tablet a33327c2d0fb44d9b3287fb08d2479de (table test-table [id=b25a3a9f1b7f4155b64ba6211a7fe6fa]) successfully deleted
W20260620 21:49:21.575116 18691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.155.129:41339 (error 108)
I20260620 21:49:21.576030 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 18900
I20260620 21:49:21.576382 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.576019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:41339 (local address 127.0.0.1:51160)
0620 21:49:21.576226 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:21.576296 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:41339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260620 21:49:21.577456 18691 catalog_manager.cc:4584] TS 2f2b3aaa4a7544c9aeaa5497c0251cd4 (127.1.155.129:41339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a33327c2d0fb44d9b3287fb08d2479de: Network error: Client connection negotiation failed: client connection to 127.1.155.129:41339: connect: Connection refused (error 111)
I20260620 21:49:21.583317 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 19034
I20260620 21:49:21.584017 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.583634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:41593 (local address 127.0.0.1:60332)
0620 21:49:21.583833 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:21.583902 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.130:41593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 21:49:21.589262 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 18672
2026-06-20T21:49:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2461 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T21:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:21Z Disabled control of system clock
I20260620 21:49:21.620680 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:39255
--webserver_interface=127.1.155.190
--webserver_port=0
--builtin_ntp_servers=127.1.155.148:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:39255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:49:21.731379 19210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:21.731745 19210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:21.731817 19210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:21.735910 19210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:21.736035 19210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:21.736066 19210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:21.736109 19210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:21.736142 19210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:21.741200 19210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:39255
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:39255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:21.742388 19210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:21.743791 19210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:21.752734 19215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:21.752727 19218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:21.752732 19216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:21.752992 19210 server_base.cc:1034] running on GCE node
I20260620 21:49:21.753556 19210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:21.754710 19210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:21.755879 19210 hybrid_clock.cc:648] HybridClock initialized: now 1781992161755859 us; error 31 us; skew 500 ppm
I20260620 21:49:21.757786 19210 webserver.cc:460] Webserver started at http://127.1.155.190:36023/ using document root <none> and password file <none>
I20260620 21:49:21.758445 19210 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:21.758512 19210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:21.758677 19210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:21.760787 19210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "d79c1ba8e6d544bf81699f27f0a953bf"
format_stamp: "Formatted at 2026-06-20 21:49:21 on dist-test-slave-5bvn"
server_key: "53a6cb109a7f37dcc73db7e7843509ea"
server_key_iv: "e39518d4c71bf309bda34b89265af5f9"
server_key_version: "encryptionkey@0"
I20260620 21:49:21.761313 19210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "d79c1ba8e6d544bf81699f27f0a953bf"
format_stamp: "Formatted at 2026-06-20 21:49:21 on dist-test-slave-5bvn"
server_key: "53a6cb109a7f37dcc73db7e7843509ea"
server_key_iv: "e39518d4c71bf309bda34b89265af5f9"
server_key_version: "encryptionkey@0"
I20260620 21:49:21.765138 19210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 21:49:21.767681 19224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:21.769171 19210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:49:21.769279 19210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "d79c1ba8e6d544bf81699f27f0a953bf"
format_stamp: "Formatted at 2026-06-20 21:49:21 on dist-test-slave-5bvn"
server_key: "53a6cb109a7f37dcc73db7e7843509ea"
server_key_iv: "e39518d4c71bf309bda34b89265af5f9"
server_key_version: "encryptionkey@0"
I20260620 21:49:21.769376 19210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:21.779088 19210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:21.779700 19210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:21.779843 19210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:21.787025 19210 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:39255
I20260620 21:49:21.787066 19276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:39255 every 8 connection(s)
I20260620 21:49:21.787976 19210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:21.791121 19277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:21.796406 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 19210
I20260620 21:49:21.796614 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:21.796921 1646 external_mini_cluster.cc:1428] Setting key 798ce13ab0551df6ed179dcdae1f23c0
I20260620 21:49:21.797168 19277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf: Bootstrap starting.
I20260620 21:49:21.799995 19277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:21.800755 19277 log.cc:824] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:21.802749 19277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf: No bootstrap required, opened a new log
I20260620 21:49:21.806362 19277 raft_consensus.cc:348] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79c1ba8e6d544bf81699f27f0a953bf" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39255 } }
I20260620 21:49:21.806641 19277 raft_consensus.cc:374] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:21.806692 19280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:21.798442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41856 (local address 127.1.155.190:39255)
0620 21:49:21.799172 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:21.799186 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:21.799225 (+ 39us) server_negotiation.cc:407] Connection header received
0620 21:49:21.799901 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:21.799918 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:21.800399 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:21.800776 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:21.801199 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.802261 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.803246 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:21.803729 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:21.803917 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:21.804828 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:21.804834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:21.804847 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:21.805250 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:21.805313 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:21.805334 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:21.805475 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:21.805597 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 21:49:21.805926 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":157,"threads_started":1}
I20260620 21:49:21.806725 19277 raft_consensus.cc:729] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d79c1ba8e6d544bf81699f27f0a953bf, State: Initialized, Role: FOLLOWER
I20260620 21:49:21.807211 19277 consensus_queue.cc:260] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79c1ba8e6d544bf81699f27f0a953bf" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39255 } }
I20260620 21:49:21.807368 19277 raft_consensus.cc:388] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:21.807430 19277 raft_consensus.cc:482] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:21.807529 19277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:21.808450 19277 raft_consensus.cc:504] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79c1ba8e6d544bf81699f27f0a953bf" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39255 } }
I20260620 21:49:21.808815 19277 leader_election.cc:302] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d79c1ba8e6d544bf81699f27f0a953bf; no voters:
I20260620 21:49:21.809114 19277 leader_election.cc:288] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:21.809242 19282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:21.809504 19282 raft_consensus.cc:686] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [term 1 LEADER]: Becoming Leader. State: Replica: d79c1ba8e6d544bf81699f27f0a953bf, State: Running, Role: LEADER
I20260620 21:49:21.809868 19282 consensus_queue.cc:237] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79c1ba8e6d544bf81699f27f0a953bf" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39255 } }
I20260620 21:49:21.810417 19277 sys_catalog.cc:564] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:21.811585 19284 sys_catalog.cc:455] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d79c1ba8e6d544bf81699f27f0a953bf. Latest consensus state: current_term: 1 leader_uuid: "d79c1ba8e6d544bf81699f27f0a953bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79c1ba8e6d544bf81699f27f0a953bf" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39255 } } }
I20260620 21:49:21.811753 19284 sys_catalog.cc:458] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:21.811615 19283 sys_catalog.cc:455] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d79c1ba8e6d544bf81699f27f0a953bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79c1ba8e6d544bf81699f27f0a953bf" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 39255 } } }
I20260620 21:49:21.811837 19283 sys_catalog.cc:458] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:21.812209 19291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:21.815376 19291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:21.821188 19291 catalog_manager.cc:1347] Generated new cluster ID: 34b763b857b8462cb6cb71fb2754482d
I20260620 21:49:21.821273 19291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:21.836813 19291 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:21.837891 19291 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:21.842679 19291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf: Generated new TSK 0
I20260620 21:49:21.843477 19291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:21.852908 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39255
--builtin_ntp_servers=127.1.155.148:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:49:21.965817 19301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:21.966159 19301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:21.966198 19301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:21.970348 19301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:21.970407 19301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:21.970463 19301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:49:21.975660 19301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:21.976761 19301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:21.978231 19301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:21.988081 19306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:21.988081 19309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:21.988194 19301 server_base.cc:1034] running on GCE node
W20260620 21:49:21.988194 19307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:21.988866 19301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:21.989552 19301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:21.990757 19301 hybrid_clock.cc:648] HybridClock initialized: now 1781992161990721 us; error 48 us; skew 500 ppm
I20260620 21:49:21.992532 19301 webserver.cc:460] Webserver started at http://127.1.155.129:38117/ using document root <none> and password file <none>
I20260620 21:49:21.993189 19301 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:21.993286 19301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:21.993531 19301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:21.995639 19301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "e1a55a16b009468f9963a9a173d5ea37"
format_stamp: "Formatted at 2026-06-20 21:49:21 on dist-test-slave-5bvn"
server_key: "5b04edbdbccabaf40261da0763d32184"
server_key_iv: "a5e42c277cfab038cbd41a527e9c4441"
server_key_version: "encryptionkey@0"
I20260620 21:49:21.996213 19301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "e1a55a16b009468f9963a9a173d5ea37"
format_stamp: "Formatted at 2026-06-20 21:49:21 on dist-test-slave-5bvn"
server_key: "5b04edbdbccabaf40261da0763d32184"
server_key_iv: "a5e42c277cfab038cbd41a527e9c4441"
server_key_version: "encryptionkey@0"
I20260620 21:49:21.999943 19301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:49:22.002383 19315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:22.003607 19301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:49:22.003762 19301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "e1a55a16b009468f9963a9a173d5ea37"
format_stamp: "Formatted at 2026-06-20 21:49:21 on dist-test-slave-5bvn"
server_key: "5b04edbdbccabaf40261da0763d32184"
server_key_iv: "a5e42c277cfab038cbd41a527e9c4441"
server_key_version: "encryptionkey@0"
I20260620 21:49:22.003885 19301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:22.023957 19301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:22.024693 19301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:22.024892 19301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:22.025475 19301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:22.026523 19301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:22.026600 19301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:22.026669 19301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:22.026721 19301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:22.035960 19301 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:41061
I20260620 21:49:22.035985 19428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:41061 every 8 connection(s)
I20260620 21:49:22.036861 19301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:49:22.039595 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 19301
I20260620 21:49:22.039726 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:49:22.040069 1646 external_mini_cluster.cc:1428] Setting key 712ec79796e090de284bf02d49f90bae
I20260620 21:49:22.042223 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39255
--builtin_ntp_servers=127.1.155.148:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:49:22.049090 19280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.038840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:59151 (local address 127.1.155.190:39255)
0620 21:49:22.038983 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:22.038986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:22.040218 (+ 1232us) server_negotiation.cc:407] Connection header received
0620 21:49:22.042077 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:22.042081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:22.042136 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:22.042239 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:22.043190 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.043717 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.046034 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.046599 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.046752 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.048479 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:22.048482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:22.048484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:22.048500 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:22.048523 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:22.048531 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:22.048602 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:22.048905 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 21:49:22.048970 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:49:22.050462 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.039237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39255 (local address 127.1.155.129:59151)
0620 21:49:22.040057 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:22.040113 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:22.041790 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:22.042423 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:22.042436 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:22.042547 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:22.043044 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.043056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.043852 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.043855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.045640 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.045674 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.046876 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.046885 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.047208 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.048047 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:49:22.048061 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:22.048132 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:49:22.048238 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:22.048635 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:22.048648 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:22.048658 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 21:49:22.048818 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 21:49:22.049206 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":152,"threads_started":1}
I20260620 21:49:22.051988 19429 heartbeater.cc:347] Connected to a master server at 127.1.155.190:39255
I20260620 21:49:22.052279 19429 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:22.052835 19429 heartbeater.cc:511] Master 127.1.155.190:39255 requested a full tablet report, sending...
I20260620 21:49:22.054581 19241 ts_manager.cc:194] Registered new tserver with Master: e1a55a16b009468f9963a9a173d5ea37 (127.1.155.129:41061)
I20260620 21:49:22.055852 19241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.155.129:59151
W20260620 21:49:22.158142 19432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:22.158444 19432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:22.158519 19432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:22.162685 19432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:22.162741 19432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:22.162799 19432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:49:22.167558 19432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:22.168542 19432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:22.169996 19432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:22.180424 19437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:22.180498 19432 server_base.cc:1034] running on GCE node
W20260620 21:49:22.180447 19438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:22.180387 19440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:22.181103 19432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:22.181761 19432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:22.182999 19432 hybrid_clock.cc:648] HybridClock initialized: now 1781992162182966 us; error 47 us; skew 500 ppm
I20260620 21:49:22.185163 19432 webserver.cc:460] Webserver started at http://127.1.155.130:34293/ using document root <none> and password file <none>
I20260620 21:49:22.185762 19432 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:22.185874 19432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:22.186100 19432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:22.188107 19432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "d10d784421734d45855febabc40449f9"
format_stamp: "Formatted at 2026-06-20 21:49:22 on dist-test-slave-5bvn"
server_key: "e4c3a45d7338558a7b17d6b32e5cb01a"
server_key_iv: "c2a8e814079c59e164ee92dc26cfd290"
server_key_version: "encryptionkey@0"
I20260620 21:49:22.188663 19432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "d10d784421734d45855febabc40449f9"
format_stamp: "Formatted at 2026-06-20 21:49:22 on dist-test-slave-5bvn"
server_key: "e4c3a45d7338558a7b17d6b32e5cb01a"
server_key_iv: "c2a8e814079c59e164ee92dc26cfd290"
server_key_version: "encryptionkey@0"
I20260620 21:49:22.192291 19432 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 21:49:22.194780 19446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:22.196027 19432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:49:22.196188 19432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "d10d784421734d45855febabc40449f9"
format_stamp: "Formatted at 2026-06-20 21:49:22 on dist-test-slave-5bvn"
server_key: "e4c3a45d7338558a7b17d6b32e5cb01a"
server_key_iv: "c2a8e814079c59e164ee92dc26cfd290"
server_key_version: "encryptionkey@0"
I20260620 21:49:22.196314 19432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:22.206118 19432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:22.206785 19432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:22.206998 19432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:22.207587 19432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:22.208570 19432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:22.208642 19432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:22.208710 19432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:22.208757 19432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:22.218987 19432 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:43779
I20260620 21:49:22.219015 19559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:43779 every 8 connection(s)
I20260620 21:49:22.219928 19432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:49:22.229955 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 19432
I20260620 21:49:22.230113 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:49:22.230126 19280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.221865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:34981 (local address 127.1.155.190:39255)
0620 21:49:22.222035 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:22.222038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:22.223155 (+ 1117us) server_negotiation.cc:407] Connection header received
0620 21:49:22.224800 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:22.224803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:22.224869 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:22.225000 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:22.226026 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.226631 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.227971 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.228222 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.228359 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.229556 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:22.229559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:22.229561 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:22.229575 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:22.229594 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:22.229601 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:22.229696 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:22.229913 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 21:49:22.229981 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:49:22.230451 1646 external_mini_cluster.cc:1428] Setting key cee98e7759127fa0513dfc9904769a30
I20260620 21:49:22.231127 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.222254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39255 (local address 127.1.155.130:34981)
0620 21:49:22.222969 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:22.223013 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:22.224546 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:22.225190 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:22.225198 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:22.225289 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:22.225797 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.225810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.226785 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.226789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.227814 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.227824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.228403 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.228407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.228594 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.229223 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:49:22.229225 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:22.229265 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:49:22.229329 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:22.229697 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:22.229709 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:22.229717 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 21:49:22.229913 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 21:49:22.230211 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":111,"threads_started":1}
I20260620 21:49:22.232566 19560 heartbeater.cc:347] Connected to a master server at 127.1.155.190:39255
I20260620 21:49:22.232731 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39255
--builtin_ntp_servers=127.1.155.148:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:49:22.232822 19560 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:22.233428 19560 heartbeater.cc:511] Master 127.1.155.190:39255 requested a full tablet report, sending...
I20260620 21:49:22.234684 19241 ts_manager.cc:194] Registered new tserver with Master: d10d784421734d45855febabc40449f9 (127.1.155.130:43779)
I20260620 21:49:22.235512 19241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.155.130:34981
W20260620 21:49:22.343247 19563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:22.343519 19563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:22.343559 19563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:22.347674 19563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:22.347736 19563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:22.347792 19563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:49:22.352834 19563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:39255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:22.353940 19563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:22.355242 19563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:22.364845 19569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:22.364827 19568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:22.364915 19563 server_base.cc:1034] running on GCE node
W20260620 21:49:22.364857 19571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:22.365545 19563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:22.366261 19563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:22.367429 19563 hybrid_clock.cc:648] HybridClock initialized: now 1781992162367395 us; error 45 us; skew 500 ppm
I20260620 21:49:22.369292 19563 webserver.cc:460] Webserver started at http://127.1.155.131:40645/ using document root <none> and password file <none>
I20260620 21:49:22.369993 19563 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:22.370062 19563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:22.370291 19563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:22.372275 19563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "efe7b2eb454b478e9a5ef15549f10615"
format_stamp: "Formatted at 2026-06-20 21:49:22 on dist-test-slave-5bvn"
server_key: "04f2267dc3bfd471ebc6adf154ef0061"
server_key_iv: "c4105be2a57142aa7043c1163720d8c1"
server_key_version: "encryptionkey@0"
I20260620 21:49:22.372900 19563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "efe7b2eb454b478e9a5ef15549f10615"
format_stamp: "Formatted at 2026-06-20 21:49:22 on dist-test-slave-5bvn"
server_key: "04f2267dc3bfd471ebc6adf154ef0061"
server_key_iv: "c4105be2a57142aa7043c1163720d8c1"
server_key_version: "encryptionkey@0"
I20260620 21:49:22.376659 19563 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:49:22.379133 19577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:22.380501 19563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:49:22.380697 19563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "efe7b2eb454b478e9a5ef15549f10615"
format_stamp: "Formatted at 2026-06-20 21:49:22 on dist-test-slave-5bvn"
server_key: "04f2267dc3bfd471ebc6adf154ef0061"
server_key_iv: "c4105be2a57142aa7043c1163720d8c1"
server_key_version: "encryptionkey@0"
I20260620 21:49:22.380820 19563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:22.397543 19563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:22.398367 19563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:22.398581 19563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:22.399202 19563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:22.400251 19563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:22.400298 19563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:22.400337 19563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:22.400353 19563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:22.409022 19563 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:45179
I20260620 21:49:22.409045 19690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:45179 every 8 connection(s)
I20260620 21:49:22.409976 19563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:49:22.419641 19280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.412011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:35075 (local address 127.1.155.190:39255)
0620 21:49:22.412172 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:22.412175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:22.413216 (+ 1041us) server_negotiation.cc:407] Connection header received
0620 21:49:22.414708 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:22.414711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:22.414768 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:22.414865 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:22.415800 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.416400 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.417576 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.417870 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.418035 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.419117 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:22.419120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:22.419122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:22.419134 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:22.419152 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:22.419160 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:22.419219 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:22.419431 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 21:49:22.419524 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 21:49:22.419776 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 19563
I20260620 21:49:22.419890 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:49:22.420189 1646 external_mini_cluster.cc:1428] Setting key 2ed80c57e995fe5bc1ec87db7ec52a4b
I20260620 21:49:22.420697 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.412390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:39255 (local address 127.1.155.131:35075)
0620 21:49:22.413071 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:22.413110 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:22.414456 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:22.415031 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:22.415037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:22.415119 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:22.415610 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.415621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.416558 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.416560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.417433 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.417442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.418088 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.418091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.418242 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.418836 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:49:22.418838 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:22.418880 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:49:22.418947 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:22.419258 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:22.419266 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:22.419271 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:49:22.419383 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 21:49:22.419701 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":116,"threads_started":1}
I20260620 21:49:22.422106 19691 heartbeater.cc:347] Connected to a master server at 127.1.155.190:39255
I20260620 21:49:22.422369 19691 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:22.423014 19691 heartbeater.cc:511] Master 127.1.155.190:39255 requested a full tablet report, sending...
I20260620 21:49:22.424065 19241 ts_manager.cc:194] Registered new tserver with Master: efe7b2eb454b478e9a5ef15549f10615 (127.1.155.131:45179)
I20260620 21:49:22.424607 19241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.155.131:35075
I20260620 21:49:22.435755 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:49:22.441651 19280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.437821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.1.155.190:39255)
0620 21:49:22.438016 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:22.438020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:22.438111 (+ 91us) server_negotiation.cc:407] Connection header received
0620 21:49:22.438209 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:22.438211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:22.438252 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:22.438314 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:22.438526 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.439055 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.439893 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.440110 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.440194 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.441175 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:22.441179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:22.441181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:22.441192 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:22.441210 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:22.441218 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:22.441281 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:22.441384 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 21:49:22.441435 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 21:49:22.444712 19241 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:49:22.447750 19241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:49:22.463701 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.457955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34836 (local address 127.1.155.130:43779)
0620 21:49:22.458569 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:22.458601 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:22.458629 (+ 28us) server_negotiation.cc:407] Connection header received
0620 21:49:22.458708 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:22.458713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:22.458925 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:22.459151 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:22.459499 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.460491 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.462175 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.462998 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.463129 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.463334 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:22.463431 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 21:49:22.463533 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":145,"threads_started":1}
I20260620 21:49:22.463717 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.457758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:43779 (local address 127.0.0.1:34836)
0620 21:49:22.458164 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:22.458215 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:22.458336 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:22.459167 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:22.459176 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:22.459205 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:22.459366 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.459371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.460621 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.460624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.462039 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.462050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.463129 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.463132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.463245 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.463282 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 21:49:22.463403 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 21:49:22.463530 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":4,"thread_start_us":109,"threads_started":1}
I20260620 21:49:22.464332 19704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.458498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:41061 (local address 127.0.0.1:44742)
0620 21:49:22.458729 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:22.458744 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:22.458823 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:22.460051 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:22.460055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:22.460074 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:22.460172 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.460178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.461520 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.461524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.462731 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.462741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.463909 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.463913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.464012 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.464036 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:49:22.464102 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 21:49:22.464196 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":59,"thread_start_us":92,"threads_started":1}
I20260620 21:49:22.464389 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.458820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:45179 (local address 127.0.0.1:58374)
0620 21:49:22.459146 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:22.459159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:22.459229 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:22.459887 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:22.459890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:22.459907 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:22.459989 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.459995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.461548 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.461552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.462566 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.462574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.464027 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.464030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.464120 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.464137 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 21:49:22.464221 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 21:49:22.464278 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":71,"thread_start_us":88,"threads_started":1}
I20260620 21:49:22.464792 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.459108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.1.155.131:45179)
0620 21:49:22.459441 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:22.459452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:22.459473 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:49:22.459549 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:22.459555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:22.459734 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:22.459890 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:22.460084 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.461397 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.462696 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.463464 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.463570 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.463753 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:22.464530 (+ 777us) server_negotiation.cc:299] Negotiation successful
0620 21:49:22.464632 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":149,"threads_started":1}
I20260620 21:49:22.464857 19705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.458704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.1.155.129:41061)
0620 21:49:22.459493 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:22.459517 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:22.459535 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:49:22.459597 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:22.459601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:22.459739 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:22.459890 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:22.460292 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.461388 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.462862 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.463611 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.463702 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.463854 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:22.464610 (+ 756us) server_negotiation.cc:299] Negotiation successful
0620 21:49:22.464704 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":97,"threads_started":1}
I20260620 21:49:22.466996 19492 tablet_service.cc:1467] Processing CreateTablet for tablet 9f965f1da9e4485d9d2fb6c787dc7787 (DEFAULT_TABLE table=test-table [id=9a93deebdec743e39c2fe709c3978d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:22.467424 19625 tablet_service.cc:1467] Processing CreateTablet for tablet 9f965f1da9e4485d9d2fb6c787dc7787 (DEFAULT_TABLE table=test-table [id=9a93deebdec743e39c2fe709c3978d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:22.468140 19492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f965f1da9e4485d9d2fb6c787dc7787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:22.468381 19625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f965f1da9e4485d9d2fb6c787dc7787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:22.468183 19363 tablet_service.cc:1467] Processing CreateTablet for tablet 9f965f1da9e4485d9d2fb6c787dc7787 (DEFAULT_TABLE table=test-table [id=9a93deebdec743e39c2fe709c3978d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:22.469074 19363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f965f1da9e4485d9d2fb6c787dc7787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:22.474613 19708 tablet_bootstrap.cc:492] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615: Bootstrap starting.
I20260620 21:49:22.476797 19710 tablet_bootstrap.cc:492] T 9f965f1da9e4485d9d2fb6c787dc7787 P e1a55a16b009468f9963a9a173d5ea37: Bootstrap starting.
I20260620 21:49:22.477349 19708 tablet_bootstrap.cc:654] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:22.478129 19708 log.cc:824] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:22.478581 19710 tablet_bootstrap.cc:654] T 9f965f1da9e4485d9d2fb6c787dc7787 P e1a55a16b009468f9963a9a173d5ea37: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:22.479223 19710 log.cc:824] T 9f965f1da9e4485d9d2fb6c787dc7787 P e1a55a16b009468f9963a9a173d5ea37: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:22.479835 19709 tablet_bootstrap.cc:492] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9: Bootstrap starting.
I20260620 21:49:22.480099 19708 tablet_bootstrap.cc:492] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615: No bootstrap required, opened a new log
I20260620 21:49:22.480290 19708 ts_tablet_manager.cc:1397] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:49:22.481714 19709 tablet_bootstrap.cc:654] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:22.482447 19709 log.cc:824] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:22.484127 19710 tablet_bootstrap.cc:492] T 9f965f1da9e4485d9d2fb6c787dc7787 P e1a55a16b009468f9963a9a173d5ea37: No bootstrap required, opened a new log
I20260620 21:49:22.484021 19708 raft_consensus.cc:348] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10d784421734d45855febabc40449f9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43779 } } peers { permanent_uuid: "e1a55a16b009468f9963a9a173d5ea37" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41061 } } peers { permanent_uuid: "efe7b2eb454b478e9a5ef15549f10615" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45179 } }
I20260620 21:49:22.484299 19708 raft_consensus.cc:374] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:22.484367 19710 ts_tablet_manager.cc:1397] T 9f965f1da9e4485d9d2fb6c787dc7787 P e1a55a16b009468f9963a9a173d5ea37: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260620 21:49:22.484398 19708 raft_consensus.cc:729] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efe7b2eb454b478e9a5ef15549f10615, State: Initialized, Role: FOLLOWER
I20260620 21:49:22.484565 19709 tablet_bootstrap.cc:492] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9: No bootstrap required, opened a new log
I20260620 21:49:22.484769 19709 ts_tablet_manager.cc:1397] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 21:49:22.484906 19708 consensus_queue.cc:260] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10d784421734d45855febabc40449f9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43779 } } peers { permanent_uuid: "e1a55a16b009468f9963a9a173d5ea37" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41061 } } peers { permanent_uuid: "efe7b2eb454b478e9a5ef15549f10615" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45179 } }
I20260620 21:49:22.485630 19691 heartbeater.cc:503] Master 127.1.155.190:39255 was elected leader, sending a full tablet report...
I20260620 21:49:22.485962 19708 ts_tablet_manager.cc:1428] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260620 21:49:22.488811 19710 raft_consensus.cc:348] T 9f965f1da9e4485d9d2fb6c787dc7787 P e1a55a16b009468f9963a9a173d5ea37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10d784421734d45855febabc40449f9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43779 } } peers { permanent_uuid: "e1a55a16b009468f9963a9a173d5ea37" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41061 } } peers { permanent_uuid: "efe7b2eb454b478e9a5ef15549f10615" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45179 } }
I20260620 21:49:22.488811 19709 raft_consensus.cc:348] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10d784421734d45855febabc40449f9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43779 } } peers { permanent_uuid: "e1a55a16b009468f9963a9a173d5ea37" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41061 } } peers { permanent_uuid: "efe7b2eb454b478e9a5ef15549f10615" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45179 } }
I20260620 21:49:22.489075 19709 raft_consensus.cc:374] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:22.489075 19710 raft_consensus.cc:374] T 9f965f1da9e4485d9d2fb6c787dc7787 P e1a55a16b009468f9963a9a173d5ea37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:22.489138 19710 raft_consensus.cc:729] T 9f965f1da9e4485d9d2fb6c787dc7787 P e1a55a16b009468f9963a9a173d5ea37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a55a16b009468f9963a9a173d5ea37, State: Initialized, Role: FOLLOWER
I20260620 21:49:22.489138 19709 raft_consensus.cc:729] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10d784421734d45855febabc40449f9, State: Initialized, Role: FOLLOWER
I20260620 21:49:22.489583 19710 consensus_queue.cc:260] T 9f965f1da9e4485d9d2fb6c787dc7787 P e1a55a16b009468f9963a9a173d5ea37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10d784421734d45855febabc40449f9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43779 } } peers { permanent_uuid: "e1a55a16b009468f9963a9a173d5ea37" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41061 } } peers { permanent_uuid: "efe7b2eb454b478e9a5ef15549f10615" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45179 } }
I20260620 21:49:22.489583 19709 consensus_queue.cc:260] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10d784421734d45855febabc40449f9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43779 } } peers { permanent_uuid: "e1a55a16b009468f9963a9a173d5ea37" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41061 } } peers { permanent_uuid: "efe7b2eb454b478e9a5ef15549f10615" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45179 } }
I20260620 21:49:22.490571 19709 ts_tablet_manager.cc:1428] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260620 21:49:22.490573 19710 ts_tablet_manager.cc:1428] T 9f965f1da9e4485d9d2fb6c787dc7787 P e1a55a16b009468f9963a9a173d5ea37: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260620 21:49:22.490938 19560 heartbeater.cc:503] Master 127.1.155.190:39255 was elected leader, sending a full tablet report...
I20260620 21:49:22.491784 19429 heartbeater.cc:503] Master 127.1.155.190:39255 was elected leader, sending a full tablet report...
I20260620 21:49:22.528168 19715 raft_consensus.cc:482] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:49:22.528366 19715 raft_consensus.cc:504] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10d784421734d45855febabc40449f9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43779 } } peers { permanent_uuid: "e1a55a16b009468f9963a9a173d5ea37" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41061 } } peers { permanent_uuid: "efe7b2eb454b478e9a5ef15549f10615" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45179 } }
I20260620 21:49:22.529448 19715 leader_election.cc:288] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a55a16b009468f9963a9a173d5ea37 (127.1.155.129:41061), efe7b2eb454b478e9a5ef15549f10615 (127.1.155.131:45179)
I20260620 21:49:22.534524 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.530270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:35691 (local address 127.1.155.131:45179)
0620 21:49:22.530406 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:22.530410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:22.530622 (+ 212us) server_negotiation.cc:407] Connection header received
0620 21:49:22.530755 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:22.530758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:22.530809 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:22.530889 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:22.531145 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.531801 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.533176 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.533857 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.533982 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.534025 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:22.534321 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 21:49:22.534400 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:49:22.534535 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.530121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:45179 (local address 127.1.155.130:35691)
0620 21:49:22.530516 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:22.530530 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:22.530627 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:22.530912 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:22.530914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:22.530929 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:22.531020 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.531040 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.531904 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.531908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.533042 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.533048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.533958 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.533961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.534116 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.534168 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 21:49:22.534253 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 21:49:22.534359 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":49,"thread_start_us":131,"threads_started":1}
I20260620 21:49:22.534524 19705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.529915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:49047 (local address 127.1.155.129:41061)
0620 21:49:22.530073 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:22.530078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:22.530095 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:49:22.530190 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:22.530193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:22.530252 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:22.530356 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:22.530784 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.531556 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.533048 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.533788 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.533964 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.534018 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:22.534321 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 21:49:22.534400 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:49:22.534559 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.529773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:41061 (local address 127.1.155.130:49047)
0620 21:49:22.529941 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:22.529959 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:22.530088 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:22.530465 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:22.530468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:22.530488 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:22.530659 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.530669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.531665 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.531668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.532888 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:22.532898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.533952 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.533957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:22.534131 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.534181 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 21:49:22.534253 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 21:49:22.534365 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":100}
I20260620 21:49:22.535269 19645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f965f1da9e4485d9d2fb6c787dc7787" candidate_uuid: "d10d784421734d45855febabc40449f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efe7b2eb454b478e9a5ef15549f10615" is_pre_election: true
I20260620 21:49:22.535389 19383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f965f1da9e4485d9d2fb6c787dc7787" candidate_uuid: "d10d784421734d45855febabc40449f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a55a16b009468f9963a9a173d5ea37" is_pre_election: true
I20260620 21:49:22.535619 19645 raft_consensus.cc:2454] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d10d784421734d45855febabc40449f9 in term 0.
I20260620 21:49:22.535735 19383 raft_consensus.cc:2454] T 9f965f1da9e4485d9d2fb6c787dc7787 P e1a55a16b009468f9963a9a173d5ea37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d10d784421734d45855febabc40449f9 in term 0.
I20260620 21:49:22.536120 19450 leader_election.cc:302] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d10d784421734d45855febabc40449f9, efe7b2eb454b478e9a5ef15549f10615; no voters:
I20260620 21:49:22.536446 19715 raft_consensus.cc:2781] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:49:22.536573 19715 raft_consensus.cc:482] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:49:22.536652 19715 raft_consensus.cc:3037] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:22.537817 19715 raft_consensus.cc:504] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10d784421734d45855febabc40449f9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43779 } } peers { permanent_uuid: "e1a55a16b009468f9963a9a173d5ea37" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41061 } } peers { permanent_uuid: "efe7b2eb454b478e9a5ef15549f10615" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45179 } }
W20260620 21:49:22.538064 19430 tablet.cc:2369] T 9f965f1da9e4485d9d2fb6c787dc7787 P e1a55a16b009468f9963a9a173d5ea37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:49:22.538232 19715 leader_election.cc:288] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [CANDIDATE]: Term 1 election: Requested vote from peers e1a55a16b009468f9963a9a173d5ea37 (127.1.155.129:41061), efe7b2eb454b478e9a5ef15549f10615 (127.1.155.131:45179)
I20260620 21:49:22.538620 19383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f965f1da9e4485d9d2fb6c787dc7787" candidate_uuid: "d10d784421734d45855febabc40449f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a55a16b009468f9963a9a173d5ea37"
I20260620 21:49:22.538645 19645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f965f1da9e4485d9d2fb6c787dc7787" candidate_uuid: "d10d784421734d45855febabc40449f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efe7b2eb454b478e9a5ef15549f10615"
I20260620 21:49:22.538772 19383 raft_consensus.cc:3037] T 9f965f1da9e4485d9d2fb6c787dc7787 P e1a55a16b009468f9963a9a173d5ea37 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:22.538780 19645 raft_consensus.cc:3037] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:22.539764 19383 raft_consensus.cc:2454] T 9f965f1da9e4485d9d2fb6c787dc7787 P e1a55a16b009468f9963a9a173d5ea37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d10d784421734d45855febabc40449f9 in term 1.
I20260620 21:49:22.539847 19645 raft_consensus.cc:2454] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d10d784421734d45855febabc40449f9 in term 1.
I20260620 21:49:22.540099 19450 leader_election.cc:302] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d10d784421734d45855febabc40449f9, e1a55a16b009468f9963a9a173d5ea37; no voters:
I20260620 21:49:22.540323 19715 raft_consensus.cc:2781] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:22.540587 19715 raft_consensus.cc:686] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [term 1 LEADER]: Becoming Leader. State: Replica: d10d784421734d45855febabc40449f9, State: Running, Role: LEADER
I20260620 21:49:22.540932 19715 consensus_queue.cc:237] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10d784421734d45855febabc40449f9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43779 } } peers { permanent_uuid: "e1a55a16b009468f9963a9a173d5ea37" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41061 } } peers { permanent_uuid: "efe7b2eb454b478e9a5ef15549f10615" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45179 } }
I20260620 21:49:22.544160 19241 catalog_manager.cc:5495] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d10d784421734d45855febabc40449f9 (127.1.155.130). New cstate: current_term: 1 leader_uuid: "d10d784421734d45855febabc40449f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10d784421734d45855febabc40449f9" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 43779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a55a16b009468f9963a9a173d5ea37" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efe7b2eb454b478e9a5ef15549f10615" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20260620 21:49:22.564697 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.561029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34838 (local address 127.1.155.130:43779)
0620 21:49:22.561174 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:22.561180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:22.561207 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:49:22.561293 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:22.561295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:22.561350 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:22.561461 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:49:22.561749 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.562325 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.563222 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:22.563444 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:22.563561 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:22.563868 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:49:22.564286 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:49:22.564422 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:22.564481 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 21:49:22.564532 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:49:22.570633 19383 raft_consensus.cc:1261] T 9f965f1da9e4485d9d2fb6c787dc7787 P e1a55a16b009468f9963a9a173d5ea37 [term 1 FOLLOWER]: Refusing update from remote peer d10d784421734d45855febabc40449f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:49:22.570673 19645 raft_consensus.cc:1261] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615 [term 1 FOLLOWER]: Refusing update from remote peer d10d784421734d45855febabc40449f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:49:22.571358 19715 consensus_queue.cc:1035] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a55a16b009468f9963a9a173d5ea37" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 41061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:49:22.571612 19718 consensus_queue.cc:1035] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe7b2eb454b478e9a5ef15549f10615" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:49:22.582404 19722 mvcc.cc:205] Tried to move back new op lower bound from 7299039897881055232 to 7299039897770397696. Current Snapshot: MvccSnapshot[applied={T|T < 7299039897881055232}]
I20260620 21:49:22.582774 19723 mvcc.cc:205] Tried to move back new op lower bound from 7299039897881055232 to 7299039897770397696. Current Snapshot: MvccSnapshot[applied={T|T < 7299039897881055232}]
I20260620 21:49:22.593061 19240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:49:22.593271 19240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:49:22.595778 19240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d79c1ba8e6d544bf81699f27f0a953bf: Sending DeleteTablet for 3 replicas of tablet 9f965f1da9e4485d9d2fb6c787dc7787
I20260620 21:49:22.596791 19625 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f965f1da9e4485d9d2fb6c787dc7787 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:22 UTC) from {username='slave'} at 127.0.0.1:58374
I20260620 21:49:22.596855 19492 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f965f1da9e4485d9d2fb6c787dc7787 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:22 UTC) from {username='slave'} at 127.0.0.1:34836
I20260620 21:49:22.597286 19731 tablet_replica.cc:330] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615: stopping tablet replica
I20260620 21:49:22.597357 19363 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f965f1da9e4485d9d2fb6c787dc7787 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:22 UTC) from {username='slave'} at 127.0.0.1:44742
I20260620 21:49:22.597606 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 19301
I20260620 21:49:22.597780 19731 raft_consensus.cc:2229] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:49:22.598165 19731 raft_consensus.cc:2258] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:49:22.597811 19732 tablet_replica.cc:330] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9: stopping tablet replica
I20260620 21:49:22.600031 19732 raft_consensus.cc:2229] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:49:22.600468 19732 raft_consensus.cc:2258] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:49:22.600469 19731 ts_tablet_manager.cc:1905] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:49:22.601935 19732 ts_tablet_manager.cc:1905] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:49:22.604074 19731 ts_tablet_manager.cc:1918] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:49:22.604281 19731 log.cc:1196] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/9f965f1da9e4485d9d2fb6c787dc7787
I20260620 21:49:22.604570 19732 ts_tablet_manager.cc:1918] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:49:22.604663 19732 log.cc:1196] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/9f965f1da9e4485d9d2fb6c787dc7787
I20260620 21:49:22.604794 19731 ts_tablet_manager.cc:1939] T 9f965f1da9e4485d9d2fb6c787dc7787 P efe7b2eb454b478e9a5ef15549f10615: Deleting consensus metadata
I20260620 21:49:22.604969 19732 ts_tablet_manager.cc:1939] T 9f965f1da9e4485d9d2fb6c787dc7787 P d10d784421734d45855febabc40449f9: Deleting consensus metadata
I20260620 21:49:22.606138 19228 catalog_manager.cc:4841] TS efe7b2eb454b478e9a5ef15549f10615 (127.1.155.131:45179): tablet 9f965f1da9e4485d9d2fb6c787dc7787 (table test-table [id=9a93deebdec743e39c2fe709c3978d1b]) successfully deleted
I20260620 21:49:22.606338 19228 catalog_manager.cc:4841] TS d10d784421734d45855febabc40449f9 (127.1.155.130:43779): tablet 9f965f1da9e4485d9d2fb6c787dc7787 (table test-table [id=9a93deebdec743e39c2fe709c3978d1b]) successfully deleted
W20260620 21:49:22.606544 19228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.155.129:41061 (error 108)
I20260620 21:49:22.606567 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 19432
I20260620 21:49:22.610796 19704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:22.610435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:41061 (local address 127.0.0.1:44744)
0620 21:49:22.610575 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:22.610643 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:41061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 21:49:22.611279 19228 catalog_manager.cc:4584] TS e1a55a16b009468f9963a9a173d5ea37 (127.1.155.129:41061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f965f1da9e4485d9d2fb6c787dc7787: Network error: Client connection negotiation failed: client connection to 127.1.155.129:41061: connect: Connection refused (error 111)
I20260620 21:49:22.612139 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 19563
I20260620 21:49:22.617980 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 19210
2026-06-20T21:49:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1028 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:22 dist-test-slave-5bvn krb5kdc[19737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:22 dist-test-slave-5bvn krb5kdc[19737](info): set up 2 sockets
Jun 20 21:49:22 dist-test-slave-5bvn krb5kdc[19737](info): commencing operation
krb5kdc: starting...
W20260620 21:49:23.705753 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:23 dist-test-slave-5bvn krb5kdc[19737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:49:23.835520 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:46189
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:46189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:23.948241 19750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:23.948518 19750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:23.948559 19750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:23.952721 19750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:23.952783 19750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:23.952803 19750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:23.952821 19750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:23.952881 19750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:23.958333 19750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:46189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:46189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:23.959411 19750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:23.960441 19750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:23.969038 19758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:23.969121 19755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:23.969133 19750 server_base.cc:1034] running on GCE node
W20260620 21:49:23.969072 19756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:23.970085 19750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:23.971179 19750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:23.972393 19750 hybrid_clock.cc:648] HybridClock initialized: now 1781992163972357 us; error 45 us; skew 500 ppm
Jun 20 21:49:23 dist-test-slave-5bvn krb5kdc[19737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992163, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:23.975306 19750 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:49:23.976138 19750 webserver.cc:460] Webserver started at http://127.1.155.190:33689/ using document root <none> and password file <none>
I20260620 21:49:23.976658 19750 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:23.976714 19750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:23.976878 19750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:23.978940 19750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "0c1988a6d8c44161999477b4019a6185"
format_stamp: "Formatted at 2026-06-20 21:49:23 on dist-test-slave-5bvn"
server_key: "566420a21c7f36cc0adb03731ce3434b"
server_key_iv: "0b1a72b1555da649f77f11b8fbc4855a"
server_key_version: "encryptionkey@0"
I20260620 21:49:23.979439 19750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "0c1988a6d8c44161999477b4019a6185"
format_stamp: "Formatted at 2026-06-20 21:49:23 on dist-test-slave-5bvn"
server_key: "566420a21c7f36cc0adb03731ce3434b"
server_key_iv: "0b1a72b1555da649f77f11b8fbc4855a"
server_key_version: "encryptionkey@0"
I20260620 21:49:23.983186 19750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:49:23.985600 19765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:23.987147 19750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:49:23.987267 19750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "0c1988a6d8c44161999477b4019a6185"
format_stamp: "Formatted at 2026-06-20 21:49:23 on dist-test-slave-5bvn"
server_key: "566420a21c7f36cc0adb03731ce3434b"
server_key_iv: "0b1a72b1555da649f77f11b8fbc4855a"
server_key_version: "encryptionkey@0"
I20260620 21:49:23.987366 19750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:24.005163 19750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:24.008368 19750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:24.008528 19750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:24.016386 19750 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:46189
I20260620 21:49:24.016412 19817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:46189 every 8 connection(s)
I20260620 21:49:24.017428 19750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:24.020293 19818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:24.022755 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 19750
I20260620 21:49:24.022904 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:24.023198 1646 external_mini_cluster.cc:1428] Setting key 7c4e0a8836551ce620f1295936c96961
I20260620 21:49:24.026871 19818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185: Bootstrap starting.
I20260620 21:49:24.029116 19818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:24.030069 19818 log.cc:824] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:24.032233 19818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185: No bootstrap required, opened a new log
Jun 20 21:49:24 dist-test-slave-5bvn krb5kdc[19737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:24.036899 19818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1988a6d8c44161999477b4019a6185" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46189 } }
I20260620 21:49:24.037175 19818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:24.037225 19818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1988a6d8c44161999477b4019a6185, State: Initialized, Role: FOLLOWER
I20260620 21:49:24.037712 19818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1988a6d8c44161999477b4019a6185" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46189 } }
I20260620 21:49:24.037901 19818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:24.037959 19818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:24.038040 19818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:24.038383 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:24.024737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.1.155.190:46189)
0620 21:49:24.025218 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:24.025242 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:24.025278 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:49:24.025962 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:24.025981 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:24.026310 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:24.026658 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:24.027460 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.028434 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.029355 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.030091 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.030261 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:24.032982 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:24.032988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:24.033004 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:24.033036 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:24.035170 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:24.035682 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:24.035688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:24.035694 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:24.035843 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:24.036161 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:24.036165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:24.036166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:24.036662 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:24.036866 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:24.037079 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 21:49:24.037371 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":140,"threads_started":1}
I20260620 21:49:24.039163 19818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1988a6d8c44161999477b4019a6185" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46189 } }
I20260620 21:49:24.039520 19818 leader_election.cc:302] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c1988a6d8c44161999477b4019a6185; no voters:
I20260620 21:49:24.039849 19818 leader_election.cc:288] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:24.039989 19823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:24.040195 19823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [term 1 LEADER]: Becoming Leader. State: Replica: 0c1988a6d8c44161999477b4019a6185, State: Running, Role: LEADER
I20260620 21:49:24.040537 19823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1988a6d8c44161999477b4019a6185" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46189 } }
I20260620 21:49:24.041146 19818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:24.041661 19823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c1988a6d8c44161999477b4019a6185. Latest consensus state: current_term: 1 leader_uuid: "0c1988a6d8c44161999477b4019a6185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1988a6d8c44161999477b4019a6185" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46189 } } }
I20260620 21:49:24.041774 19823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:24.042342 19829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:24.042315 19824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c1988a6d8c44161999477b4019a6185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1988a6d8c44161999477b4019a6185" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 46189 } } }
I20260620 21:49:24.042529 19824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:24.045619 19829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:24.051689 19829 catalog_manager.cc:1347] Generated new cluster ID: 3b758b9fb4244bc6bb6bd247fcd01086
I20260620 21:49:24.051780 19829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:24.064215 19829 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:24.065184 19829 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:24.072465 19829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185: Generated new TSK 0
I20260620 21:49:24.073258 19829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:49:24.110261 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46189
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:24.223799 19844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:24.224135 19844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:24.224213 19844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:24.228331 19844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:24.228451 19844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:24.228546 19844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:49:24.233860 19844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:24.235003 19844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:24.236035 19844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:24.245981 19849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:24.245972 19852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:24.246208 19844 server_base.cc:1034] running on GCE node
W20260620 21:49:24.245972 19850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:24.246714 19844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:24.247402 19844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:24.248577 19844 hybrid_clock.cc:648] HybridClock initialized: now 1781992164248558 us; error 29 us; skew 500 ppm
Jun 20 21:49:24 dist-test-slave-5bvn krb5kdc[19737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:24.257138 19844 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:49:24.258376 19844 webserver.cc:460] Webserver started at http://127.1.155.129:45717/ using document root <none> and password file <none>
I20260620 21:49:24.258929 19844 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:24.258996 19844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:24.259166 19844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:24.261449 19844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "0e341a6c89754053a452b07bf1f7aed9"
format_stamp: "Formatted at 2026-06-20 21:49:24 on dist-test-slave-5bvn"
server_key: "2a002e026fb67b78ea2bb4d5f505d18c"
server_key_iv: "a7dfb045f42833c7de8973a3387ffc7f"
server_key_version: "encryptionkey@0"
I20260620 21:49:24.262053 19844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "0e341a6c89754053a452b07bf1f7aed9"
format_stamp: "Formatted at 2026-06-20 21:49:24 on dist-test-slave-5bvn"
server_key: "2a002e026fb67b78ea2bb4d5f505d18c"
server_key_iv: "a7dfb045f42833c7de8973a3387ffc7f"
server_key_version: "encryptionkey@0"
I20260620 21:49:24.265936 19844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:49:24.268642 19859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:24.270072 19844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:49:24.270208 19844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "0e341a6c89754053a452b07bf1f7aed9"
format_stamp: "Formatted at 2026-06-20 21:49:24 on dist-test-slave-5bvn"
server_key: "2a002e026fb67b78ea2bb4d5f505d18c"
server_key_iv: "a7dfb045f42833c7de8973a3387ffc7f"
server_key_version: "encryptionkey@0"
I20260620 21:49:24.270330 19844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:24.284359 19844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:24.287981 19844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:24.288200 19844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:24.288935 19844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:24.289945 19844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:24.289995 19844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:24.290037 19844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:24.290052 19844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:24.299388 19844 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:42527
I20260620 21:49:24.299404 19972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:42527 every 8 connection(s)
I20260620 21:49:24.300287 19844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:49:24.307539 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 19844
I20260620 21:49:24.307763 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:49:24.308064 1646 external_mini_cluster.cc:1428] Setting key 002a0428459c5152c0019effdf2ffba6
Jun 20 21:49:24 dist-test-slave-5bvn krb5kdc[19737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:24.314497 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:24.302096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:40355 (local address 127.1.155.190:46189)
0620 21:49:24.302240 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:24.302244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:24.303384 (+ 1140us) server_negotiation.cc:407] Connection header received
0620 21:49:24.304296 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:24.304299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:24.304357 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:24.304486 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:24.305627 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.306236 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.307374 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.307612 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.307733 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:24.310526 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:24.310533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:24.310539 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:24.310572 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:24.312077 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:24.312763 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:24.312766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:24.312769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:24.312856 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:24.313282 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:24.313284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:24.313285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:24.313462 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:24.313540 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:24.314260 (+ 720us) server_negotiation.cc:299] Negotiation successful
0620 21:49:24.314372 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:49:24.315896 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:24.302516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46189 (local address 127.1.155.129:40355)
0620 21:49:24.303205 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:24.303245 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:24.304072 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:24.304652 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:24.304658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:24.305013 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:24.305464 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:24.305476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.306365 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.306369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:24.307237 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:24.307246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.307768 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.307772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:24.307974 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:24.308658 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:24.308675 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:24.310360 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:24.312208 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:24.312217 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:24.312236 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:24.312635 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:24.312963 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:24.312969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:24.312974 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:24.313165 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:24.313576 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:24.313585 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:24.313954 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 21:49:24.314196 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 21:49:24.314584 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":120,"threads_started":1}
I20260620 21:49:24.317349 19973 heartbeater.cc:347] Connected to a master server at 127.1.155.190:46189
I20260620 21:49:24.317618 19973 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:24.318259 19973 heartbeater.cc:511] Master 127.1.155.190:46189 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
I20260620 21:49:24.320024 19782 ts_manager.cc:194] Registered new tserver with Master: 0e341a6c89754053a452b07bf1f7aed9 (127.1.155.129:42527)
I20260620 21:49:24.321247 19782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:40355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:49:24.341725 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46189
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:24.453562 19978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:24.453907 19978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:24.454005 19978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:24.458230 19978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:24.458292 19978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:24.458386 19978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:49:24.463842 19978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:24.464984 19978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:24.466068 19978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:24.476015 19986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:24.476164 19978 server_base.cc:1034] running on GCE node
W20260620 21:49:24.476063 19984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:24.476025 19983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:24.476863 19978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:24.477519 19978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:24.478720 19978 hybrid_clock.cc:648] HybridClock initialized: now 1781992164478662 us; error 71 us; skew 500 ppm
Jun 20 21:49:24 dist-test-slave-5bvn krb5kdc[19737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:24.481451 19978 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:49:24.482438 19978 webserver.cc:460] Webserver started at http://127.1.155.130:35899/ using document root <none> and password file <none>
I20260620 21:49:24.483007 19978 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:24.483103 19978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:24.483340 19978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:24.485425 19978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237"
format_stamp: "Formatted at 2026-06-20 21:49:24 on dist-test-slave-5bvn"
server_key: "d355b79cd267784d5b5eb2e39890a353"
server_key_iv: "5eaf91f44d56e8be4fe1af8819af7e61"
server_key_version: "encryptionkey@0"
I20260620 21:49:24.486097 19978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237"
format_stamp: "Formatted at 2026-06-20 21:49:24 on dist-test-slave-5bvn"
server_key: "d355b79cd267784d5b5eb2e39890a353"
server_key_iv: "5eaf91f44d56e8be4fe1af8819af7e61"
server_key_version: "encryptionkey@0"
I20260620 21:49:24.489964 19978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:49:24.492411 19993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:24.493659 19978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:49:24.493757 19978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237"
format_stamp: "Formatted at 2026-06-20 21:49:24 on dist-test-slave-5bvn"
server_key: "d355b79cd267784d5b5eb2e39890a353"
server_key_iv: "5eaf91f44d56e8be4fe1af8819af7e61"
server_key_version: "encryptionkey@0"
I20260620 21:49:24.493927 19978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:24.510636 19978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:24.514093 19978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:24.514328 19978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:24.515017 19978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:24.515964 19978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:24.516014 19978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:24.516113 19978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:24.516160 19978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:24.525154 19978 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:32951
I20260620 21:49:24.525168 20106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:32951 every 8 connection(s)
I20260620 21:49:24.526087 19978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:49:24.528957 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 19978
I20260620 21:49:24.529078 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:49:24.529372 1646 external_mini_cluster.cc:1428] Setting key f97f9db6f84d5267717498c9b2ba8979
Jun 20 21:49:24 dist-test-slave-5bvn krb5kdc[19737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
I20260620 21:49:24.542017 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:24.528042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:44643 (local address 127.1.155.190:46189)
0620 21:49:24.528202 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:24.528206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:24.529036 (+ 830us) server_negotiation.cc:407] Connection header received
0620 21:49:24.530587 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:24.530590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:24.530677 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:24.530804 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:24.532089 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.532929 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.534613 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.534827 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.534913 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:24.537803 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:24.537808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:24.537813 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:24.537870 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:24.539683 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:24.540233 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:24.540237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:24.540240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:24.540327 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:24.540742 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:24.540746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:24.540748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:24.540953 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:24.541042 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:24.541653 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 21:49:24.541873 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
I20260620 21:49:24.543002 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:24.528366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46189 (local address 127.1.155.130:44643)
0620 21:49:24.528858 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:24.528901 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:24.530348 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:24.531030 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:24.531039 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:24.531415 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:24.531929 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:24.531942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.533090 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.533092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:24.534474 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:24.534489 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.535004 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.535009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:24.535269 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:24.535989 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:24.536006 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:24.537581 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:24.539810 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:24.539815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:24.539829 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:24.540084 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:24.540431 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:24.540433 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:24.540434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:24.540624 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:24.541063 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:24.541068 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:24.541329 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 21:49:24.541517 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 21:49:24.541965 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":108,"threads_started":1}
I20260620 21:49:24.544168 20107 heartbeater.cc:347] Connected to a master server at 127.1.155.190:46189
I20260620 21:49:24.544452 20107 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:24.544971 20107 heartbeater.cc:511] Master 127.1.155.190:46189 requested a full tablet report, sending...
I20260620 21:49:24.546154 19782 ts_manager.cc:194] Registered new tserver with Master: 8aa64fefa96a4bf6ba4a6aa6eba05237 (127.1.155.130:32951)
I20260620 21:49:24.546911 19782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:44643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:49:24.561321 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46189
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:24.672523 20112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:24.672823 20112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:24.672895 20112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:24.677209 20112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:24.677297 20112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:24.677385 20112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:49:24.682612 20112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:46189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:24.683904 20112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:24.685027 20112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:24.694871 20120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:24.694944 20118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:24.694880 20117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:24.695035 20112 server_base.cc:1034] running on GCE node
I20260620 21:49:24.695513 20112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:24.696177 20112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:24.697368 20112 hybrid_clock.cc:648] HybridClock initialized: now 1781992164697340 us; error 40 us; skew 500 ppm
Jun 20 21:49:24 dist-test-slave-5bvn krb5kdc[19737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:24.700239 20112 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:49:24.701161 20112 webserver.cc:460] Webserver started at http://127.1.155.131:40611/ using document root <none> and password file <none>
I20260620 21:49:24.701678 20112 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:24.701742 20112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:24.701982 20112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:24.703986 20112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "cff2d5962e2c45d4ac8db8239856680a"
format_stamp: "Formatted at 2026-06-20 21:49:24 on dist-test-slave-5bvn"
server_key: "6e19800c4045f576187015f514999a9b"
server_key_iv: "3517a07589e39fce464bb37374d8c8aa"
server_key_version: "encryptionkey@0"
I20260620 21:49:24.704530 20112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "cff2d5962e2c45d4ac8db8239856680a"
format_stamp: "Formatted at 2026-06-20 21:49:24 on dist-test-slave-5bvn"
server_key: "6e19800c4045f576187015f514999a9b"
server_key_iv: "3517a07589e39fce464bb37374d8c8aa"
server_key_version: "encryptionkey@0"
I20260620 21:49:24.708398 20112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:49:24.710973 20127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:24.712284 20112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:49:24.712430 20112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "cff2d5962e2c45d4ac8db8239856680a"
format_stamp: "Formatted at 2026-06-20 21:49:24 on dist-test-slave-5bvn"
server_key: "6e19800c4045f576187015f514999a9b"
server_key_iv: "3517a07589e39fce464bb37374d8c8aa"
server_key_version: "encryptionkey@0"
I20260620 21:49:24.712556 20112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:24.742445 20112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:24.745666 20112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:24.746011 20112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:24.746745 20112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:24.747596 20112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:24.747644 20112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:24.747684 20112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:24.747735 20112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:24.758160 20112 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:34067
I20260620 21:49:24.758176 20240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:34067 every 8 connection(s)
I20260620 21:49:24.759079 20112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:49:24.768431 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 20112
I20260620 21:49:24.768566 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
Jun 20 21:49:24 dist-test-slave-5bvn krb5kdc[19737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:24.768904 1646 external_mini_cluster.cc:1428] Setting key 4433aa266a6fdf5c325a3fdf3eb3b0b1
I20260620 21:49:24.773445 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:24.760918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:32839 (local address 127.1.155.190:46189)
0620 21:49:24.761060 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:24.761064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:24.761957 (+ 893us) server_negotiation.cc:407] Connection header received
0620 21:49:24.763239 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:24.763243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:24.763321 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:24.763429 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:24.764612 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.765148 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.766488 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.766716 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.766845 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:24.769690 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:24.769693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:24.769696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:24.769724 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:24.771549 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:24.772064 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:24.772066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:24.772068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:24.772141 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:24.772441 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:24.772443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:24.772444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:24.772610 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:24.772681 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:24.773156 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 21:49:24.773259 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:49:24.774349 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:24.761329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:46189 (local address 127.1.155.131:32839)
0620 21:49:24.761779 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:24.761822 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:24.763019 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:24.763600 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:24.763607 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:24.763966 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:24.764450 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:24.764464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.765299 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.765303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:24.766362 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:24.766372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.766888 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.766892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:24.767068 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:24.767646 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:24.767662 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:24.769457 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:24.771656 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:24.771661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:24.771673 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:24.771970 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:24.772225 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:24.772227 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:24.772229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:24.772361 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:24.772697 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:24.772702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:24.772952 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 21:49:24.773109 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 21:49:24.773382 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20260620 21:49:24.775549 20241 heartbeater.cc:347] Connected to a master server at 127.1.155.190:46189
I20260620 21:49:24.775796 20241 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:24.776240 20241 heartbeater.cc:511] Master 127.1.155.190:46189 requested a full tablet report, sending...
I20260620 21:49:24.777235 19782 ts_manager.cc:194] Registered new tserver with Master: cff2d5962e2c45d4ac8db8239856680a (127.1.155.131:34067)
I20260620 21:49:24.777808 19782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:32839
I20260620 21:49:24.782883 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:49:24.793098 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:24.784865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.1.155.190:46189)
0620 21:49:24.785035 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:24.785040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:24.785203 (+ 163us) server_negotiation.cc:407] Connection header received
0620 21:49:24.785289 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:24.785291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:24.785329 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:24.785390 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:24.786110 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.786725 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.787591 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.787812 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.787903 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:24.789483 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:24.789486 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:24.789487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:24.789512 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:24.791172 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:24.791682 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:24.791684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:24.791686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:24.791758 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:24.792145 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:24.792147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:24.792148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:24.792366 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:24.792483 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:24.792762 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 21:49:24.792910 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 21:49:24.796344 19782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:49:24.799634 19782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:49:24.814118 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:24.808350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:42527 (local address 127.0.0.1:36578)
0620 21:49:24.808834 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:24.808891 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:24.809060 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:24.809721 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:24.809728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:24.809752 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:24.809941 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:24.809951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.811465 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.811468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:24.812384 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:24.812392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.813499 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.813503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:24.813686 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:24.813749 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 21:49:24.813886 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 21:49:24.813998 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":22,"thread_start_us":93,"threads_started":1}
I20260620 21:49:24.814965 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:24.808552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.1.155.129:42527)
0620 21:49:24.808931 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:24.808950 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:24.808977 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:49:24.809267 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:24.809273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:24.809532 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:24.810106 (+ 574us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:24.810267 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.811301 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.812531 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.813368 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.814532 (+ 1164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:24.814687 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:24.814734 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 21:49:24.814826 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":97,"threads_started":1}
I20260620 21:49:24.815369 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:24.809262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:32951 (local address 127.0.0.1:53656)
0620 21:49:24.809546 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:24.809566 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:24.809684 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:24.810705 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:24.810708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:24.810719 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:24.810799 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:24.810804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.812368 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.812372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:24.813639 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:24.813650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.814895 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.814899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:24.814997 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:24.815025 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:49:24.815123 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 21:49:24.815209 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":104,"thread_start_us":110,"threads_started":1}
I20260620 21:49:24.815426 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:24.809452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.1.155.130:32951)
0620 21:49:24.810003 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:24.810019 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:24.810046 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:49:24.810129 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:24.810135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:24.810371 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:24.810639 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:24.810920 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.812205 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.813782 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.814742 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.814898 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:24.815058 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:24.815160 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 21:49:24.815258 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":116,"threads_started":1}
I20260620 21:49:24.816430 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:24.810171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.1.155.131:34067)
0620 21:49:24.810744 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:24.810759 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:24.810785 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:49:24.810871 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:24.810879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:24.811076 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:24.811260 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:24.811457 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.812640 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.814561 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.815838 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.815961 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:24.816085 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:24.816205 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 21:49:24.816289 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":94,"threads_started":1}
I20260620 21:49:24.816581 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:24.809581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:34067 (local address 127.0.0.1:39520)
0620 21:49:24.810511 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:24.810525 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:24.810608 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:24.811244 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:24.811247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:24.811263 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:24.811351 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:24.811358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.813050 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.813054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:24.814417 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:24.814430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:24.815981 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:24.815984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:24.816076 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:24.816098 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:49:24.816175 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 21:49:24.816244 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260620 21:49:24.818424 20041 tablet_service.cc:1467] Processing CreateTablet for tablet f91ea7e52d994b93b95df379c9b0a6d6 (DEFAULT_TABLE table=test-table [id=f907ae8cc4b5427187a592bfeeb74684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:24.818387 19907 tablet_service.cc:1467] Processing CreateTablet for tablet f91ea7e52d994b93b95df379c9b0a6d6 (DEFAULT_TABLE table=test-table [id=f907ae8cc4b5427187a592bfeeb74684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:24.819108 20175 tablet_service.cc:1467] Processing CreateTablet for tablet f91ea7e52d994b93b95df379c9b0a6d6 (DEFAULT_TABLE table=test-table [id=f907ae8cc4b5427187a592bfeeb74684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:24.819645 20041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f91ea7e52d994b93b95df379c9b0a6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:24.819645 19907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f91ea7e52d994b93b95df379c9b0a6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:24.820003 20175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f91ea7e52d994b93b95df379c9b0a6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:24.826181 20258 tablet_bootstrap.cc:492] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a: Bootstrap starting.
I20260620 21:49:24.827661 20259 tablet_bootstrap.cc:492] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9: Bootstrap starting.
I20260620 21:49:24.828540 20258 tablet_bootstrap.cc:654] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:24.829813 20258 log.cc:824] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:24.829928 20259 tablet_bootstrap.cc:654] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:24.829921 20260 tablet_bootstrap.cc:492] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237: Bootstrap starting.
I20260620 21:49:24.830754 20259 log.cc:824] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:24.831911 20258 tablet_bootstrap.cc:492] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a: No bootstrap required, opened a new log
I20260620 21:49:24.832101 20258 ts_tablet_manager.cc:1397] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 21:49:24.832774 20259 tablet_bootstrap.cc:492] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9: No bootstrap required, opened a new log
I20260620 21:49:24.832971 20259 ts_tablet_manager.cc:1397] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 21:49:24.833170 20260 tablet_bootstrap.cc:654] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:24.834044 20260 log.cc:824] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:24.835855 20260 tablet_bootstrap.cc:492] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237: No bootstrap required, opened a new log
I20260620 21:49:24.836089 20260 ts_tablet_manager.cc:1397] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:49:24.836185 20258 raft_consensus.cc:348] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e341a6c89754053a452b07bf1f7aed9" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42527 } } peers { permanent_uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 32951 } } peers { permanent_uuid: "cff2d5962e2c45d4ac8db8239856680a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34067 } }
I20260620 21:49:24.836483 20258 raft_consensus.cc:374] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:24.836541 20258 raft_consensus.cc:729] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff2d5962e2c45d4ac8db8239856680a, State: Initialized, Role: FOLLOWER
I20260620 21:49:24.836997 20258 consensus_queue.cc:260] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e341a6c89754053a452b07bf1f7aed9" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42527 } } peers { permanent_uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 32951 } } peers { permanent_uuid: "cff2d5962e2c45d4ac8db8239856680a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34067 } }
I20260620 21:49:24.837375 20259 raft_consensus.cc:348] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e341a6c89754053a452b07bf1f7aed9" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42527 } } peers { permanent_uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 32951 } } peers { permanent_uuid: "cff2d5962e2c45d4ac8db8239856680a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34067 } }
I20260620 21:49:24.837636 20259 raft_consensus.cc:374] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:24.837687 20259 raft_consensus.cc:729] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e341a6c89754053a452b07bf1f7aed9, State: Initialized, Role: FOLLOWER
I20260620 21:49:24.838018 20241 heartbeater.cc:503] Master 127.1.155.190:46189 was elected leader, sending a full tablet report...
I20260620 21:49:24.838093 20258 ts_tablet_manager.cc:1428] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260620 21:49:24.838174 20259 consensus_queue.cc:260] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e341a6c89754053a452b07bf1f7aed9" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42527 } } peers { permanent_uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 32951 } } peers { permanent_uuid: "cff2d5962e2c45d4ac8db8239856680a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34067 } }
I20260620 21:49:24.838806 19973 heartbeater.cc:503] Master 127.1.155.190:46189 was elected leader, sending a full tablet report...
I20260620 21:49:24.839365 20259 ts_tablet_manager.cc:1428] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:49:24.843377 20260 raft_consensus.cc:348] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e341a6c89754053a452b07bf1f7aed9" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42527 } } peers { permanent_uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 32951 } } peers { permanent_uuid: "cff2d5962e2c45d4ac8db8239856680a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34067 } }
I20260620 21:49:24.843724 20260 raft_consensus.cc:374] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:24.843777 20260 raft_consensus.cc:729] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aa64fefa96a4bf6ba4a6aa6eba05237, State: Initialized, Role: FOLLOWER
I20260620 21:49:24.844269 20260 consensus_queue.cc:260] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e341a6c89754053a452b07bf1f7aed9" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42527 } } peers { permanent_uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 32951 } } peers { permanent_uuid: "cff2d5962e2c45d4ac8db8239856680a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34067 } }
I20260620 21:49:24.844977 20107 heartbeater.cc:503] Master 127.1.155.190:46189 was elected leader, sending a full tablet report...
I20260620 21:49:24.845221 20260 ts_tablet_manager.cc:1428] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237: Time spent starting tablet: real 0.009s user 0.001s sys 0.006s
W20260620 21:49:25.010102 20242 tablet.cc:2369] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:49:25.027521 20108 tablet.cc:2369] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:49:25.037075 20265 raft_consensus.cc:482] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:49:25.037377 20265 raft_consensus.cc:504] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e341a6c89754053a452b07bf1f7aed9" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42527 } } peers { permanent_uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 32951 } } peers { permanent_uuid: "cff2d5962e2c45d4ac8db8239856680a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34067 } }
I20260620 21:49:25.038563 20265 leader_election.cc:288] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8aa64fefa96a4bf6ba4a6aa6eba05237 (127.1.155.130:32951), cff2d5962e2c45d4ac8db8239856680a (127.1.155.131:34067)
I20260620 21:49:25.044179 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:25.039547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:46069 (local address 127.1.155.131:34067)
0620 21:49:25.039702 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:25.039707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:25.039793 (+ 86us) server_negotiation.cc:407] Connection header received
0620 21:49:25.039909 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:25.039911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:25.039994 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:25.040075 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:25.040318 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.041021 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.042843 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.043501 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.043635 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:25.043678 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:25.043968 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 21:49:25.044047 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 21:49:25.044179 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:25.039045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:36139 (local address 127.1.155.130:32951)
0620 21:49:25.039187 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:25.039191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:25.039405 (+ 214us) server_negotiation.cc:407] Connection header received
0620 21:49:25.039569 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:25.039574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:25.039637 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:25.039737 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:25.040041 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.040854 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.042863 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.043497 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.043624 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:25.043678 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:25.043983 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 21:49:25.044055 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:49:25.044231 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:25.038962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:32951 (local address 127.1.155.129:36139)
0620 21:49:25.039275 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:25.039314 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:25.039434 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:25.039771 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:25.039776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:25.039806 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:25.039959 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:25.039965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.041029 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.041032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:25.042702 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:25.042714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.043632 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.043636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:25.043853 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:25.043874 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:49:25.043943 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 21:49:25.044041 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":244,"thread_start_us":138,"threads_started":1}
I20260620 21:49:25.044230 20268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:25.039401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:34067 (local address 127.1.155.129:46069)
0620 21:49:25.039689 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:25.039729 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:25.039825 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:25.040101 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:25.040103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:25.040118 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:25.040207 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:25.040211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.041113 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.041116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:25.042710 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:25.042715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.043632 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.043636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:25.043815 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:25.043843 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:49:25.043928 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 21:49:25.044045 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":188,"thread_start_us":84,"threads_started":1}
I20260620 21:49:25.045049 20061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f91ea7e52d994b93b95df379c9b0a6d6" candidate_uuid: "0e341a6c89754053a452b07bf1f7aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237" is_pre_election: true
I20260620 21:49:25.045049 20195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f91ea7e52d994b93b95df379c9b0a6d6" candidate_uuid: "0e341a6c89754053a452b07bf1f7aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff2d5962e2c45d4ac8db8239856680a" is_pre_election: true
I20260620 21:49:25.045387 20061 raft_consensus.cc:2454] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e341a6c89754053a452b07bf1f7aed9 in term 0.
I20260620 21:49:25.045387 20195 raft_consensus.cc:2454] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e341a6c89754053a452b07bf1f7aed9 in term 0.
I20260620 21:49:25.045912 19861 leader_election.cc:302] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e341a6c89754053a452b07bf1f7aed9, 8aa64fefa96a4bf6ba4a6aa6eba05237; no voters:
I20260620 21:49:25.046172 20265 raft_consensus.cc:2781] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:49:25.046257 20265 raft_consensus.cc:482] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:49:25.046299 20265 raft_consensus.cc:3037] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:25.047490 20265 raft_consensus.cc:504] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e341a6c89754053a452b07bf1f7aed9" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42527 } } peers { permanent_uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 32951 } } peers { permanent_uuid: "cff2d5962e2c45d4ac8db8239856680a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34067 } }
I20260620 21:49:25.047868 20265 leader_election.cc:288] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [CANDIDATE]: Term 1 election: Requested vote from peers 8aa64fefa96a4bf6ba4a6aa6eba05237 (127.1.155.130:32951), cff2d5962e2c45d4ac8db8239856680a (127.1.155.131:34067)
I20260620 21:49:25.048303 20195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f91ea7e52d994b93b95df379c9b0a6d6" candidate_uuid: "0e341a6c89754053a452b07bf1f7aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff2d5962e2c45d4ac8db8239856680a"
I20260620 21:49:25.048303 20061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f91ea7e52d994b93b95df379c9b0a6d6" candidate_uuid: "0e341a6c89754053a452b07bf1f7aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237"
I20260620 21:49:25.048442 20195 raft_consensus.cc:3037] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:25.048442 20061 raft_consensus.cc:3037] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:25.049710 20195 raft_consensus.cc:2454] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e341a6c89754053a452b07bf1f7aed9 in term 1.
I20260620 21:49:25.049710 20061 raft_consensus.cc:2454] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e341a6c89754053a452b07bf1f7aed9 in term 1.
I20260620 21:49:25.050158 19861 leader_election.cc:302] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e341a6c89754053a452b07bf1f7aed9, cff2d5962e2c45d4ac8db8239856680a; no voters:
I20260620 21:49:25.050379 20265 raft_consensus.cc:2781] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:25.050635 20265 raft_consensus.cc:686] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [term 1 LEADER]: Becoming Leader. State: Replica: 0e341a6c89754053a452b07bf1f7aed9, State: Running, Role: LEADER
I20260620 21:49:25.050980 20265 consensus_queue.cc:237] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e341a6c89754053a452b07bf1f7aed9" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42527 } } peers { permanent_uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 32951 } } peers { permanent_uuid: "cff2d5962e2c45d4ac8db8239856680a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34067 } }
W20260620 21:49:25.051560 19974 tablet.cc:2369] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:49:25.054356 19781 catalog_manager.cc:5495] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e341a6c89754053a452b07bf1f7aed9 (127.1.155.129). New cstate: current_term: 1 leader_uuid: "0e341a6c89754053a452b07bf1f7aed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e341a6c89754053a452b07bf1f7aed9" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff2d5962e2c45d4ac8db8239856680a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34067 } health_report { overall_health: UNKNOWN } } }
I20260620 21:49:25.118841 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:25.115101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.1.155.129:42527)
0620 21:49:25.115287 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:25.115291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:25.115310 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:49:25.115405 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:25.115408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:25.115462 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:25.115579 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:49:25.115844 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.116368 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.117305 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.117519 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.117633 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:25.117983 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:49:25.118413 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:49:25.118574 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:25.118658 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 21:49:25.118701 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 21:49:25.125031 20061 raft_consensus.cc:1261] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237 [term 1 FOLLOWER]: Refusing update from remote peer 0e341a6c89754053a452b07bf1f7aed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:49:25.125023 20195 raft_consensus.cc:1261] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a [term 1 FOLLOWER]: Refusing update from remote peer 0e341a6c89754053a452b07bf1f7aed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:49:25.125890 20265 consensus_queue.cc:1035] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8aa64fefa96a4bf6ba4a6aa6eba05237" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:49:25.126174 20269 consensus_queue.cc:1035] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff2d5962e2c45d4ac8db8239856680a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:49:25.134891 20273 mvcc.cc:205] Tried to move back new op lower bound from 7299039908343255040 to 7299039908051791872. Current Snapshot: MvccSnapshot[applied={T|T < 7299039908343255040}]
I20260620 21:49:25.137362 20274 mvcc.cc:205] Tried to move back new op lower bound from 7299039908343255040 to 7299039908051791872. Current Snapshot: MvccSnapshot[applied={T|T < 7299039908343255040}]
I20260620 21:49:25.142912 19780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:49:25.143118 19780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:49:25.145984 19780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c1988a6d8c44161999477b4019a6185: Sending DeleteTablet for 3 replicas of tablet f91ea7e52d994b93b95df379c9b0a6d6
I20260620 21:49:25.146801 19907 tablet_service.cc:1514] Processing DeleteTablet for tablet f91ea7e52d994b93b95df379c9b0a6d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:25 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:36578
I20260620 21:49:25.146911 20041 tablet_service.cc:1514] Processing DeleteTablet for tablet f91ea7e52d994b93b95df379c9b0a6d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:25 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:53656
I20260620 21:49:25.147441 20283 tablet_replica.cc:330] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237: stopping tablet replica
I20260620 21:49:25.147372 20175 tablet_service.cc:1514] Processing DeleteTablet for tablet f91ea7e52d994b93b95df379c9b0a6d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:25 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:39520
I20260620 21:49:25.147799 20283 raft_consensus.cc:2229] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:49:25.147883 20282 tablet_replica.cc:330] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9: stopping tablet replica
I20260620 21:49:25.148149 20283 raft_consensus.cc:2258] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:49:25.148181 20282 raft_consensus.cc:2229] T f91ea7e52d994b93b95df379c9b0a6d6 P 0e341a6c89754053a452b07bf1f7aed9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:49:25.148283 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 19844
I20260620 21:49:25.150775 20283 ts_tablet_manager.cc:1905] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:49:25.153038 20283 ts_tablet_manager.cc:1918] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:49:25.153177 20283 log.cc:1196] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/f91ea7e52d994b93b95df379c9b0a6d6
I20260620 21:49:25.153426 20284 tablet_replica.cc:330] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a: stopping tablet replica
I20260620 21:49:25.153512 20283 ts_tablet_manager.cc:1939] T f91ea7e52d994b93b95df379c9b0a6d6 P 8aa64fefa96a4bf6ba4a6aa6eba05237: Deleting consensus metadata
I20260620 21:49:25.153787 20284 raft_consensus.cc:2229] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:49:25.154153 20284 raft_consensus.cc:2258] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:49:25.155308 19767 catalog_manager.cc:4841] TS 8aa64fefa96a4bf6ba4a6aa6eba05237 (127.1.155.130:32951): tablet f91ea7e52d994b93b95df379c9b0a6d6 (table test-table [id=f907ae8cc4b5427187a592bfeeb74684]) successfully deleted
I20260620 21:49:25.155618 20284 ts_tablet_manager.cc:1905] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 21:49:25.156026 19767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.155.129:42527 (error 108)
I20260620 21:49:25.156934 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 19978
I20260620 21:49:25.157511 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:25.156812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:42527 (local address 127.0.0.1:36600)
0620 21:49:25.157309 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:25.157395 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:42527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377}
I20260620 21:49:25.158450 20284 ts_tablet_manager.cc:1918] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:49:25.158573 20284 log.cc:1196] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/f91ea7e52d994b93b95df379c9b0a6d6
I20260620 21:49:25.158892 20284 ts_tablet_manager.cc:1939] T f91ea7e52d994b93b95df379c9b0a6d6 P cff2d5962e2c45d4ac8db8239856680a: Deleting consensus metadata
W20260620 21:49:25.162518 19767 catalog_manager.cc:4584] TS 0e341a6c89754053a452b07bf1f7aed9 (127.1.155.129:42527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f91ea7e52d994b93b95df379c9b0a6d6: Network error: Client connection negotiation failed: client connection to 127.1.155.129:42527: connect: Connection refused (error 111)
I20260620 21:49:25.162808 19767 catalog_manager.cc:4841] TS cff2d5962e2c45d4ac8db8239856680a (127.1.155.131:34067): tablet f91ea7e52d994b93b95df379c9b0a6d6 (table test-table [id=f907ae8cc4b5427187a592bfeeb74684]) successfully deleted
I20260620 21:49:25.163697 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 20112
I20260620 21:49:25.169555 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 19750
2026-06-20T21:49:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2551 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T21:49:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:25Z Disabled control of system clock
I20260620 21:49:25.199813 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:40863
--webserver_interface=127.1.155.190
--webserver_port=0
--builtin_ntp_servers=127.1.155.148:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:40863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:49:25.309764 20289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:25.310046 20289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:25.310082 20289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:25.314132 20289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:25.314194 20289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:25.314210 20289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:25.314227 20289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:25.314244 20289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:25.319078 20289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:40863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:40863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:25.320152 20289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:25.321535 20289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:25.330236 20294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:25.330267 20297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:25.330245 20295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:25.330559 20289 server_base.cc:1034] running on GCE node
I20260620 21:49:25.331080 20289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:25.332250 20289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:25.333444 20289 hybrid_clock.cc:648] HybridClock initialized: now 1781992165333420 us; error 40 us; skew 500 ppm
I20260620 21:49:25.335436 20289 webserver.cc:460] Webserver started at http://127.1.155.190:35605/ using document root <none> and password file <none>
I20260620 21:49:25.336030 20289 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:25.336126 20289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:25.336354 20289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:25.338491 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "b7ccbbee00564394b96aa66fd0a89a89"
format_stamp: "Formatted at 2026-06-20 21:49:25 on dist-test-slave-5bvn"
server_key: "7954b428c103ba195d2207ada4db3034"
server_key_iv: "b471c115086e87ecf5ecb8c321c9986d"
server_key_version: "encryptionkey@0"
I20260620 21:49:25.339046 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "b7ccbbee00564394b96aa66fd0a89a89"
format_stamp: "Formatted at 2026-06-20 21:49:25 on dist-test-slave-5bvn"
server_key: "7954b428c103ba195d2207ada4db3034"
server_key_iv: "b471c115086e87ecf5ecb8c321c9986d"
server_key_version: "encryptionkey@0"
I20260620 21:49:25.342705 20289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:49:25.345098 20303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:25.346387 20289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:49:25.346486 20289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "b7ccbbee00564394b96aa66fd0a89a89"
format_stamp: "Formatted at 2026-06-20 21:49:25 on dist-test-slave-5bvn"
server_key: "7954b428c103ba195d2207ada4db3034"
server_key_iv: "b471c115086e87ecf5ecb8c321c9986d"
server_key_version: "encryptionkey@0"
I20260620 21:49:25.346567 20289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:25.357338 20289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:25.358142 20289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:25.358385 20289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:25.366044 20289 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:40863
I20260620 21:49:25.366104 20355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:40863 every 8 connection(s)
I20260620 21:49:25.367033 20289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:25.370170 20356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:25.375517 20356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89: Bootstrap starting.
I20260620 21:49:25.375837 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 20289
I20260620 21:49:25.375981 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:25.376259 1646 external_mini_cluster.cc:1428] Setting key 537e9e02eb29903377082d878ef11a1e
I20260620 21:49:25.378301 20356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:25.379179 20356 log.cc:824] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:25.381383 20356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89: No bootstrap required, opened a new log
I20260620 21:49:25.386030 20356 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ccbbee00564394b96aa66fd0a89a89" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 40863 } }
I20260620 21:49:25.386300 20356 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:25.386360 20356 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ccbbee00564394b96aa66fd0a89a89, State: Initialized, Role: FOLLOWER
I20260620 21:49:25.386868 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:25.377651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35164 (local address 127.1.155.190:40863)
0620 21:49:25.378385 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:25.378405 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:25.378448 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:49:25.379320 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:25.379344 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:25.379817 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:25.380200 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:25.380626 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.382091 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.382977 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.383681 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.383846 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:25.384707 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:25.384716 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:25.384736 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:25.385332 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:25.385398 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:25.385426 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:25.385525 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:25.385630 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 21:49:25.385924 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":160,"threads_started":1}
I20260620 21:49:25.386828 20356 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ccbbee00564394b96aa66fd0a89a89" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 40863 } }
I20260620 21:49:25.386973 20356 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:25.387027 20356 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:25.387122 20356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:25.387985 20356 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ccbbee00564394b96aa66fd0a89a89" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 40863 } }
I20260620 21:49:25.388332 20356 leader_election.cc:302] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7ccbbee00564394b96aa66fd0a89a89; no voters:
I20260620 21:49:25.388633 20356 leader_election.cc:288] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:25.388736 20361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:25.388993 20361 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [term 1 LEADER]: Becoming Leader. State: Replica: b7ccbbee00564394b96aa66fd0a89a89, State: Running, Role: LEADER
I20260620 21:49:25.389603 20361 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ccbbee00564394b96aa66fd0a89a89" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 40863 } }
I20260620 21:49:25.389974 20356 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:25.390784 20363 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7ccbbee00564394b96aa66fd0a89a89. Latest consensus state: current_term: 1 leader_uuid: "b7ccbbee00564394b96aa66fd0a89a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ccbbee00564394b96aa66fd0a89a89" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 40863 } } }
I20260620 21:49:25.390923 20363 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:25.391130 20362 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7ccbbee00564394b96aa66fd0a89a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ccbbee00564394b96aa66fd0a89a89" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 40863 } } }
I20260620 21:49:25.391223 20362 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:25.391340 20370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:25.394634 20370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:25.400255 20370 catalog_manager.cc:1347] Generated new cluster ID: 8f024aec3deb4824be0b8e14d9531abf
I20260620 21:49:25.400326 20370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:25.425129 20370 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:25.426461 20370 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:25.434444 20370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89: Generated new TSK 0
I20260620 21:49:25.435276 20370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:25.444916 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:40863
--builtin_ntp_servers=127.1.155.148:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:49:25.554297 20380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:25.554597 20380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:25.554667 20380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:25.558768 20380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:25.558853 20380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:25.558945 20380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:49:25.563746 20380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:40863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:25.564867 20380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:25.566357 20380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:25.576248 20385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:25.576462 20380 server_base.cc:1034] running on GCE node
W20260620 21:49:25.576251 20388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:25.576248 20386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:25.577026 20380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:25.577749 20380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:25.578967 20380 hybrid_clock.cc:648] HybridClock initialized: now 1781992165578936 us; error 43 us; skew 500 ppm
I20260620 21:49:25.581065 20380 webserver.cc:460] Webserver started at http://127.1.155.129:40901/ using document root <none> and password file <none>
I20260620 21:49:25.581653 20380 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:25.581768 20380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:25.582046 20380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:25.584193 20380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3"
format_stamp: "Formatted at 2026-06-20 21:49:25 on dist-test-slave-5bvn"
server_key: "fa87d592e954edb23242dd802458e71e"
server_key_iv: "e183aa12a9996e60d2c7e03c701b59bb"
server_key_version: "encryptionkey@0"
I20260620 21:49:25.584797 20380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3"
format_stamp: "Formatted at 2026-06-20 21:49:25 on dist-test-slave-5bvn"
server_key: "fa87d592e954edb23242dd802458e71e"
server_key_iv: "e183aa12a9996e60d2c7e03c701b59bb"
server_key_version: "encryptionkey@0"
I20260620 21:49:25.588891 20380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 21:49:25.592382 20394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:25.594046 20380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 21:49:25.594184 20380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3"
format_stamp: "Formatted at 2026-06-20 21:49:25 on dist-test-slave-5bvn"
server_key: "fa87d592e954edb23242dd802458e71e"
server_key_iv: "e183aa12a9996e60d2c7e03c701b59bb"
server_key_version: "encryptionkey@0"
I20260620 21:49:25.594327 20380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:25.610042 20380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:25.610770 20380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:25.610981 20380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:25.611573 20380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:25.612596 20380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:25.612679 20380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:25.612764 20380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:25.612828 20380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:25.622154 20380 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:34751
I20260620 21:49:25.622172 20507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:34751 every 8 connection(s)
I20260620 21:49:25.623083 20380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:49:25.631699 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 20380
I20260620 21:49:25.631861 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:49:25.632140 1646 external_mini_cluster.cc:1428] Setting key d0adffb8c37ec7981868f7aa0e72cd34
I20260620 21:49:25.633150 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:25.625032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:49649 (local address 127.1.155.190:40863)
0620 21:49:25.625240 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:25.625246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:25.626409 (+ 1163us) server_negotiation.cc:407] Connection header received
0620 21:49:25.627872 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:25.627875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:25.627923 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:25.628003 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:25.628968 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.629517 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.630739 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.630966 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.631054 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:25.632452 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:25.632458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:25.632463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:25.632480 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:25.632499 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:25.632508 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:25.632596 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:25.632811 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 21:49:25.632906 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 21:49:25.634343 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:40863
--builtin_ntp_servers=127.1.155.148:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:49:25.634363 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:25.625442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:40863 (local address 127.1.155.129:49649)
0620 21:49:25.626234 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:25.626279 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:25.627633 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:25.628181 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:25.628187 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:25.628274 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:25.628775 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:25.628787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.629668 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.629671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:25.630607 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:25.630617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.631112 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.631115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:25.631252 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:25.632125 (+ 873us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:49:25.632127 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:25.632169 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:49:25.632269 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:25.632613 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:25.632622 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:25.632628 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:49:25.632780 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 21:49:25.633255 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":113,"threads_started":1}
I20260620 21:49:25.636351 20508 heartbeater.cc:347] Connected to a master server at 127.1.155.190:40863
I20260620 21:49:25.636618 20508 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:25.637187 20508 heartbeater.cc:511] Master 127.1.155.190:40863 requested a full tablet report, sending...
I20260620 21:49:25.638888 20320 ts_manager.cc:194] Registered new tserver with Master: 7cd008e1efbb4b50bbb44c2b8ebe2bb3 (127.1.155.129:34751)
I20260620 21:49:25.640089 20320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.155.129:49649
W20260620 21:49:25.751847 20511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:25.752148 20511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:25.752220 20511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:25.756363 20511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:25.756465 20511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:25.756559 20511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:49:25.761504 20511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:40863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:25.762653 20511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:25.764017 20511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:25.773804 20517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:25.773938 20516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:25.774065 20511 server_base.cc:1034] running on GCE node
W20260620 21:49:25.774226 20519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:25.774609 20511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:25.775250 20511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:25.776414 20511 hybrid_clock.cc:648] HybridClock initialized: now 1781992165776384 us; error 45 us; skew 500 ppm
I20260620 21:49:25.778347 20511 webserver.cc:460] Webserver started at http://127.1.155.130:33313/ using document root <none> and password file <none>
I20260620 21:49:25.778930 20511 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:25.779003 20511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:25.779225 20511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:25.781265 20511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "dd56f1c4f94c4245b3ce4996e2803382"
format_stamp: "Formatted at 2026-06-20 21:49:25 on dist-test-slave-5bvn"
server_key: "be13c5305c24fe5f1b7e8c47ddc4fbdd"
server_key_iv: "74c4d630c7a447ca19c17bdef7f0eecf"
server_key_version: "encryptionkey@0"
I20260620 21:49:25.781919 20511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "dd56f1c4f94c4245b3ce4996e2803382"
format_stamp: "Formatted at 2026-06-20 21:49:25 on dist-test-slave-5bvn"
server_key: "be13c5305c24fe5f1b7e8c47ddc4fbdd"
server_key_iv: "74c4d630c7a447ca19c17bdef7f0eecf"
server_key_version: "encryptionkey@0"
I20260620 21:49:25.785678 20511 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:49:25.788214 20525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:25.789599 20511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:49:25.789747 20511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "dd56f1c4f94c4245b3ce4996e2803382"
format_stamp: "Formatted at 2026-06-20 21:49:25 on dist-test-slave-5bvn"
server_key: "be13c5305c24fe5f1b7e8c47ddc4fbdd"
server_key_iv: "74c4d630c7a447ca19c17bdef7f0eecf"
server_key_version: "encryptionkey@0"
I20260620 21:49:25.789917 20511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:25.806504 20511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:25.807260 20511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:25.807485 20511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:25.808111 20511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:25.809326 20511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:25.809432 20511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:25.809506 20511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:25.809571 20511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:25.818773 20511 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:41659
I20260620 21:49:25.818790 20638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:41659 every 8 connection(s)
I20260620 21:49:25.819690 20511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:49:25.821733 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 20511
I20260620 21:49:25.821889 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:49:25.822208 1646 external_mini_cluster.cc:1428] Setting key 9439ef1a760ed4753154a66df7eed1f7
I20260620 21:49:25.824545 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:40863
--builtin_ntp_servers=127.1.155.148:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:49:25.830929 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:25.821738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:36935 (local address 127.1.155.190:40863)
0620 21:49:25.821932 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:25.821937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:25.823222 (+ 1285us) server_negotiation.cc:407] Connection header received
0620 21:49:25.825155 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:25.825158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:25.825211 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:25.825290 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:25.826513 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.827380 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.828744 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.828994 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.829119 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:25.830379 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:25.830383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:25.830385 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:25.830400 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:25.830428 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:25.830436 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:25.830527 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:25.830704 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 21:49:25.830775 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 21:49:25.831976 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:25.822247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:40863 (local address 127.1.155.130:36935)
0620 21:49:25.823073 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:25.823117 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:25.824858 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:25.825515 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:25.825527 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:25.825645 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:25.826351 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:25.826367 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.827570 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.827574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:25.828596 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:25.828607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:25.829166 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:25.829170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:25.829355 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:25.830072 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:49:25.830074 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:25.830118 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:49:25.830191 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:25.830529 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:25.830542 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:25.830548 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:49:25.830688 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 21:49:25.831109 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":127,"threads_started":1}
I20260620 21:49:25.833662 20639 heartbeater.cc:347] Connected to a master server at 127.1.155.190:40863
I20260620 21:49:25.833994 20639 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:25.834784 20639 heartbeater.cc:511] Master 127.1.155.190:40863 requested a full tablet report, sending...
I20260620 21:49:25.836151 20320 ts_manager.cc:194] Registered new tserver with Master: dd56f1c4f94c4245b3ce4996e2803382 (127.1.155.130:41659)
I20260620 21:49:25.836786 20320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.155.130:36935
W20260620 21:49:25.941977 20642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:25.942231 20642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:25.942269 20642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:25.946355 20642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:25.946404 20642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:25.946509 20642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:49:25.951678 20642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:40863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:25.952764 20642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:25.954209 20642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:25.963717 20647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:25.963732 20648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:25.963707 20650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:25.964051 20642 server_base.cc:1034] running on GCE node
I20260620 21:49:25.964440 20642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:25.965112 20642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:25.966274 20642 hybrid_clock.cc:648] HybridClock initialized: now 1781992165966259 us; error 27 us; skew 500 ppm
I20260620 21:49:25.968156 20642 webserver.cc:460] Webserver started at http://127.1.155.131:33207/ using document root <none> and password file <none>
I20260620 21:49:25.968732 20642 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:25.968801 20642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:25.969049 20642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:25.971087 20642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "055801db18b8425083011285766b96b1"
format_stamp: "Formatted at 2026-06-20 21:49:25 on dist-test-slave-5bvn"
server_key: "dd4dc58ac9e6585d7d8e0442fe942ada"
server_key_iv: "578dbbf71d30b95040f939cf1f9566dc"
server_key_version: "encryptionkey@0"
I20260620 21:49:25.971647 20642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "055801db18b8425083011285766b96b1"
format_stamp: "Formatted at 2026-06-20 21:49:25 on dist-test-slave-5bvn"
server_key: "dd4dc58ac9e6585d7d8e0442fe942ada"
server_key_iv: "578dbbf71d30b95040f939cf1f9566dc"
server_key_version: "encryptionkey@0"
I20260620 21:49:25.975443 20642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:49:25.977991 20656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:25.979350 20642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:49:25.979457 20642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "055801db18b8425083011285766b96b1"
format_stamp: "Formatted at 2026-06-20 21:49:25 on dist-test-slave-5bvn"
server_key: "dd4dc58ac9e6585d7d8e0442fe942ada"
server_key_iv: "578dbbf71d30b95040f939cf1f9566dc"
server_key_version: "encryptionkey@0"
I20260620 21:49:25.979554 20642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:25.990785 20642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:25.991487 20642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:25.991631 20642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:25.992221 20642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:25.993181 20642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:25.993228 20642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:25.993268 20642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:25.993283 20642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:26.002393 20642 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:40323
I20260620 21:49:26.002467 20769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:40323 every 8 connection(s)
I20260620 21:49:26.003296 20642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:49:26.012231 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 20642
I20260620 21:49:26.012374 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:49:26.012408 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:26.005110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:45249 (local address 127.1.155.190:40863)
0620 21:49:26.005257 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:26.005261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:26.006340 (+ 1079us) server_negotiation.cc:407] Connection header received
0620 21:49:26.007741 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:26.007744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:26.007802 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:26.007922 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:26.008804 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.009335 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.010506 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.010736 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.010867 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:26.011899 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:26.011903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:26.011906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:26.011925 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:26.011943 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:26.011951 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:26.012035 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:26.012204 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 21:49:26.012264 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:49:26.012713 1646 external_mini_cluster.cc:1428] Setting key f767efa0e3cc727757a42e68d4be00f0
I20260620 21:49:26.013413 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:26.005494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:40863 (local address 127.1.155.131:45249)
0620 21:49:26.006183 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:26.006222 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:26.007516 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:26.008092 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:26.008099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:26.008177 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:26.008643 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:26.008655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.009472 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.009476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:26.010367 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:26.010376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.010900 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.010904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:26.011071 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:26.011636 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:49:26.011638 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:26.011675 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:49:26.011734 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:26.012050 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:26.012061 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:26.012069 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 21:49:26.012203 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 21:49:26.012479 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":103,"threads_started":1}
I20260620 21:49:26.014496 20770 heartbeater.cc:347] Connected to a master server at 127.1.155.190:40863
I20260620 21:49:26.014742 20770 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:26.015255 20770 heartbeater.cc:511] Master 127.1.155.190:40863 requested a full tablet report, sending...
I20260620 21:49:26.016494 20320 ts_manager.cc:194] Registered new tserver with Master: 055801db18b8425083011285766b96b1 (127.1.155.131:40323)
I20260620 21:49:26.017025 20320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.155.131:45249
I20260620 21:49:26.027339 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:49:26.033556 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:26.029363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.1.155.190:40863)
0620 21:49:26.029572 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:26.029577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:26.029705 (+ 128us) server_negotiation.cc:407] Connection header received
0620 21:49:26.029875 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:26.029879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:26.029958 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:26.030056 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:26.030345 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.030875 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.031758 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.031980 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.032096 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:26.033013 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:26.033016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:26.033018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:26.033030 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:26.033049 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:26.033056 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:26.033197 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:26.033359 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 21:49:26.033400 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 21:49:26.036600 20320 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:49:26.039590 20320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:49:26.055372 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:26.049136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38516 (local address 127.1.155.130:41659)
0620 21:49:26.050331 (+ 1195us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:26.050345 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:26.050374 (+ 29us) server_negotiation.cc:407] Connection header received
0620 21:49:26.050453 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:26.050460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:26.050683 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:26.050852 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:26.051212 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.052461 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.054123 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.054657 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.054796 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:26.055000 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:26.055092 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 21:49:26.055196 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068,"thread_start_us":148,"threads_started":1}
I20260620 21:49:26.055361 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:26.049081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:34751 (local address 127.0.0.1:38198)
0620 21:49:26.049837 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:26.049849 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:26.049965 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:26.051110 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:26.051113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:26.051134 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:26.051412 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:26.051417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.052390 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.052396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:26.053951 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:26.053963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.054930 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.054933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:26.055004 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:26.055021 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 21:49:26.055063 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 21:49:26.055193 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":281,"spinlock_wait_cycles":2816,"thread_start_us":152,"threads_started":1}
I20260620 21:49:26.055583 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:26.049594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:40323 (local address 127.0.0.1:43472)
0620 21:49:26.050190 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:26.050203 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:26.050284 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:26.051083 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:26.051089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:26.051110 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:26.051233 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:26.051240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.052620 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.052623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:26.053753 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:26.053762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.054876 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.054881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:26.054976 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:26.055003 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:49:26.055053 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 21:49:26.055455 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":310,"thread_start_us":80,"threads_started":1}
I20260620 21:49:26.055670 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:26.050214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.1.155.131:40323)
0620 21:49:26.050515 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:26.050592 (+ 77us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:26.050617 (+ 25us) server_negotiation.cc:407] Connection header received
0620 21:49:26.050684 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:26.050687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:26.050855 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:26.051049 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:26.051365 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.052473 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.053905 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.054728 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.055178 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:26.055356 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:26.055416 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 21:49:26.055512 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":101,"threads_started":1}
I20260620 21:49:26.055365 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:26.048861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:41659 (local address 127.0.0.1:38516)
0620 21:49:26.049663 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:26.049717 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:26.049914 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:26.050891 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:26.050898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:26.050924 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:26.051082 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:26.051091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.052729 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.052732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:26.053986 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:26.054000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.054795 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.054798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:26.054913 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:26.054949 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 21:49:26.055055 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 21:49:26.055183 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":27,"thread_start_us":84,"threads_started":1}
I20260620 21:49:26.056226 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:26.049689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.1.155.129:34751)
0620 21:49:26.050686 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:26.050698 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:26.050719 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:49:26.050794 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:26.050799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:26.050961 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:26.051203 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:26.051528 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.052252 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.054083 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.054798 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.055755 (+ 957us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:26.055945 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:26.055990 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 21:49:26.056077 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":903,"thread_start_us":156,"threads_started":1}
I20260620 21:49:26.058727 20442 tablet_service.cc:1467] Processing CreateTablet for tablet 64ce5999c0a149e4bd76c5a81728524d (DEFAULT_TABLE table=test-table [id=ec9038bee89948febda2a80b19de3501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:26.058776 20704 tablet_service.cc:1467] Processing CreateTablet for tablet 64ce5999c0a149e4bd76c5a81728524d (DEFAULT_TABLE table=test-table [id=ec9038bee89948febda2a80b19de3501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:26.058804 20573 tablet_service.cc:1467] Processing CreateTablet for tablet 64ce5999c0a149e4bd76c5a81728524d (DEFAULT_TABLE table=test-table [id=ec9038bee89948febda2a80b19de3501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:26.059705 20442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ce5999c0a149e4bd76c5a81728524d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:26.059759 20704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ce5999c0a149e4bd76c5a81728524d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:26.059759 20573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ce5999c0a149e4bd76c5a81728524d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:26.066097 20787 tablet_bootstrap.cc:492] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1: Bootstrap starting.
I20260620 21:49:26.067835 20789 tablet_bootstrap.cc:492] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3: Bootstrap starting.
I20260620 21:49:26.069025 20787 tablet_bootstrap.cc:654] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:26.069612 20789 tablet_bootstrap.cc:654] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:26.069828 20787 log.cc:824] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:26.070211 20788 tablet_bootstrap.cc:492] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382: Bootstrap starting.
I20260620 21:49:26.070506 20789 log.cc:824] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:26.071935 20787 tablet_bootstrap.cc:492] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1: No bootstrap required, opened a new log
I20260620 21:49:26.072153 20787 ts_tablet_manager.cc:1397] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:49:26.072309 20788 tablet_bootstrap.cc:654] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:26.072515 20789 tablet_bootstrap.cc:492] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3: No bootstrap required, opened a new log
I20260620 21:49:26.072706 20789 ts_tablet_manager.cc:1397] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 21:49:26.073184 20788 log.cc:824] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:26.075192 20788 tablet_bootstrap.cc:492] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382: No bootstrap required, opened a new log
I20260620 21:49:26.075392 20788 ts_tablet_manager.cc:1397] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 21:49:26.076047 20789 raft_consensus.cc:348] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd56f1c4f94c4245b3ce4996e2803382" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41659 } } peers { permanent_uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34751 } } peers { permanent_uuid: "055801db18b8425083011285766b96b1" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40323 } }
I20260620 21:49:26.076242 20789 raft_consensus.cc:374] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:26.076274 20789 raft_consensus.cc:729] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cd008e1efbb4b50bbb44c2b8ebe2bb3, State: Initialized, Role: FOLLOWER
I20260620 21:49:26.076740 20789 consensus_queue.cc:260] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd56f1c4f94c4245b3ce4996e2803382" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41659 } } peers { permanent_uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34751 } } peers { permanent_uuid: "055801db18b8425083011285766b96b1" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40323 } }
I20260620 21:49:26.076766 20787 raft_consensus.cc:348] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd56f1c4f94c4245b3ce4996e2803382" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41659 } } peers { permanent_uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34751 } } peers { permanent_uuid: "055801db18b8425083011285766b96b1" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40323 } }
I20260620 21:49:26.077009 20787 raft_consensus.cc:374] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:26.077064 20787 raft_consensus.cc:729] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055801db18b8425083011285766b96b1, State: Initialized, Role: FOLLOWER
I20260620 21:49:26.077507 20508 heartbeater.cc:503] Master 127.1.155.190:40863 was elected leader, sending a full tablet report...
I20260620 21:49:26.077764 20789 ts_tablet_manager.cc:1428] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 21:49:26.078187 20787 consensus_queue.cc:260] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd56f1c4f94c4245b3ce4996e2803382" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41659 } } peers { permanent_uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34751 } } peers { permanent_uuid: "055801db18b8425083011285766b96b1" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40323 } }
I20260620 21:49:26.078943 20770 heartbeater.cc:503] Master 127.1.155.190:40863 was elected leader, sending a full tablet report...
I20260620 21:49:26.079120 20787 ts_tablet_manager.cc:1428] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 21:49:26.079500 20788 raft_consensus.cc:348] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd56f1c4f94c4245b3ce4996e2803382" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41659 } } peers { permanent_uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34751 } } peers { permanent_uuid: "055801db18b8425083011285766b96b1" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40323 } }
I20260620 21:49:26.079733 20788 raft_consensus.cc:374] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:26.079787 20788 raft_consensus.cc:729] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd56f1c4f94c4245b3ce4996e2803382, State: Initialized, Role: FOLLOWER
I20260620 21:49:26.080194 20788 consensus_queue.cc:260] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd56f1c4f94c4245b3ce4996e2803382" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41659 } } peers { permanent_uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34751 } } peers { permanent_uuid: "055801db18b8425083011285766b96b1" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40323 } }
I20260620 21:49:26.080848 20639 heartbeater.cc:503] Master 127.1.155.190:40863 was elected leader, sending a full tablet report...
I20260620 21:49:26.081039 20788 ts_tablet_manager.cc:1428] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:49:26.088996 20793 raft_consensus.cc:482] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:49:26.089250 20793 raft_consensus.cc:504] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd56f1c4f94c4245b3ce4996e2803382" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41659 } } peers { permanent_uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34751 } } peers { permanent_uuid: "055801db18b8425083011285766b96b1" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40323 } }
I20260620 21:49:26.090667 20793 leader_election.cc:288] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd56f1c4f94c4245b3ce4996e2803382 (127.1.155.130:41659), 055801db18b8425083011285766b96b1 (127.1.155.131:40323)
I20260620 21:49:26.094774 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:26.090840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:41659 (local address 127.1.155.129:51417)
0620 21:49:26.091073 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:26.091088 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:26.091193 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:26.091592 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:26.091595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:26.091609 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:26.091759 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:26.091770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.092537 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.092543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:26.093527 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:26.093539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.094335 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.094339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:26.094453 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:26.094476 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:49:26.094555 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 21:49:26.094632 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":44}
I20260620 21:49:26.094823 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:26.090896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:51417 (local address 127.1.155.130:41659)
0620 21:49:26.091063 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:26.091068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:26.091161 (+ 93us) server_negotiation.cc:407] Connection header received
0620 21:49:26.091303 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:26.091306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:26.091370 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:26.091450 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:26.091875 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.092424 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.093686 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.094230 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.094343 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:26.094379 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:26.094600 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 21:49:26.094703 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 21:49:26.094887 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:26.090899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:40323 (local address 127.1.155.129:38701)
0620 21:49:26.091302 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:26.091314 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:26.091449 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:26.091733 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:26.091736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:26.091754 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:26.091864 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:26.091870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.092615 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.092618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:26.093522 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:26.093531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.094422 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.094425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:26.094500 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:26.094531 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 21:49:26.094676 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 21:49:26.094732 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":2,"thread_start_us":134,"threads_started":1}
I20260620 21:49:26.094902 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:26.091064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:38701 (local address 127.1.155.131:40323)
0620 21:49:26.091202 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:26.091210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:26.091423 (+ 213us) server_negotiation.cc:407] Connection header received
0620 21:49:26.091556 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:26.091559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:26.091600 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:26.091699 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:26.091994 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.092463 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.093634 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.094182 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.094298 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:26.094352 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:26.094712 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 21:49:26.094788 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:49:26.095460 20724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ce5999c0a149e4bd76c5a81728524d" candidate_uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055801db18b8425083011285766b96b1" is_pre_election: true
I20260620 21:49:26.095605 20593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ce5999c0a149e4bd76c5a81728524d" candidate_uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd56f1c4f94c4245b3ce4996e2803382" is_pre_election: true
I20260620 21:49:26.095736 20724 raft_consensus.cc:2454] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cd008e1efbb4b50bbb44c2b8ebe2bb3 in term 0.
I20260620 21:49:26.095892 20593 raft_consensus.cc:2454] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cd008e1efbb4b50bbb44c2b8ebe2bb3 in term 0.
I20260620 21:49:26.096302 20395 leader_election.cc:302] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055801db18b8425083011285766b96b1, 7cd008e1efbb4b50bbb44c2b8ebe2bb3; no voters:
I20260620 21:49:26.096604 20793 raft_consensus.cc:2781] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:49:26.096684 20793 raft_consensus.cc:482] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:49:26.096715 20793 raft_consensus.cc:3037] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:26.097792 20793 raft_consensus.cc:504] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd56f1c4f94c4245b3ce4996e2803382" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41659 } } peers { permanent_uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34751 } } peers { permanent_uuid: "055801db18b8425083011285766b96b1" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40323 } }
I20260620 21:49:26.098189 20793 leader_election.cc:288] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [CANDIDATE]: Term 1 election: Requested vote from peers dd56f1c4f94c4245b3ce4996e2803382 (127.1.155.130:41659), 055801db18b8425083011285766b96b1 (127.1.155.131:40323)
I20260620 21:49:26.098572 20593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ce5999c0a149e4bd76c5a81728524d" candidate_uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd56f1c4f94c4245b3ce4996e2803382"
I20260620 21:49:26.098579 20724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ce5999c0a149e4bd76c5a81728524d" candidate_uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055801db18b8425083011285766b96b1"
I20260620 21:49:26.098701 20724 raft_consensus.cc:3037] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:26.098699 20593 raft_consensus.cc:3037] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:26.099979 20593 raft_consensus.cc:2454] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cd008e1efbb4b50bbb44c2b8ebe2bb3 in term 1.
I20260620 21:49:26.099974 20724 raft_consensus.cc:2454] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cd008e1efbb4b50bbb44c2b8ebe2bb3 in term 1.
I20260620 21:49:26.100271 20396 leader_election.cc:302] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cd008e1efbb4b50bbb44c2b8ebe2bb3, dd56f1c4f94c4245b3ce4996e2803382; no voters:
I20260620 21:49:26.100427 20793 raft_consensus.cc:2781] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:26.100646 20793 raft_consensus.cc:686] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [term 1 LEADER]: Becoming Leader. State: Replica: 7cd008e1efbb4b50bbb44c2b8ebe2bb3, State: Running, Role: LEADER
I20260620 21:49:26.100957 20793 consensus_queue.cc:237] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd56f1c4f94c4245b3ce4996e2803382" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41659 } } peers { permanent_uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34751 } } peers { permanent_uuid: "055801db18b8425083011285766b96b1" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40323 } }
I20260620 21:49:26.104076 20318 catalog_manager.cc:5495] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cd008e1efbb4b50bbb44c2b8ebe2bb3 (127.1.155.129). New cstate: current_term: 1 leader_uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd56f1c4f94c4245b3ce4996e2803382" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cd008e1efbb4b50bbb44c2b8ebe2bb3" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 34751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "055801db18b8425083011285766b96b1" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40323 } health_report { overall_health: UNKNOWN } } }
I20260620 21:49:26.122195 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:26.118367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.1.155.129:34751)
0620 21:49:26.118515 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:26.118518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:26.118531 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:49:26.118598 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:26.118600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:26.118651 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:26.118756 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:49:26.119001 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.119536 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.120418 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:26.120634 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:26.120722 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:26.121185 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:49:26.121607 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:49:26.121728 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:26.121890 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 21:49:26.121978 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 21:49:26.124233 20509 tablet.cc:2369] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:49:26.128625 20593 raft_consensus.cc:1261] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382 [term 1 FOLLOWER]: Refusing update from remote peer 7cd008e1efbb4b50bbb44c2b8ebe2bb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:49:26.128625 20724 raft_consensus.cc:1261] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1 [term 1 FOLLOWER]: Refusing update from remote peer 7cd008e1efbb4b50bbb44c2b8ebe2bb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:49:26.129364 20793 consensus_queue.cc:1035] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd56f1c4f94c4245b3ce4996e2803382" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:49:26.129556 20797 consensus_queue.cc:1035] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "055801db18b8425083011285766b96b1" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 40323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:49:26.136227 20800 mvcc.cc:205] Tried to move back new op lower bound from 7299039912453955584 to 7299039912352129024. Current Snapshot: MvccSnapshot[applied={T|T < 7299039912453955584}]
I20260620 21:49:26.137036 20801 mvcc.cc:205] Tried to move back new op lower bound from 7299039912453955584 to 7299039912352129024. Current Snapshot: MvccSnapshot[applied={T|T < 7299039912453955584}]
I20260620 21:49:26.142664 20802 mvcc.cc:205] Tried to move back new op lower bound from 7299039912453955584 to 7299039912352129024. Current Snapshot: MvccSnapshot[applied={T|T < 7299039912453955584}]
I20260620 21:49:26.148273 20319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:49:26.148468 20319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:49:26.151412 20319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7ccbbee00564394b96aa66fd0a89a89: Sending DeleteTablet for 3 replicas of tablet 64ce5999c0a149e4bd76c5a81728524d
I20260620 21:49:26.152411 20704 tablet_service.cc:1514] Processing DeleteTablet for tablet 64ce5999c0a149e4bd76c5a81728524d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:26 UTC) from {username='slave'} at 127.0.0.1:43472
I20260620 21:49:26.152478 20573 tablet_service.cc:1514] Processing DeleteTablet for tablet 64ce5999c0a149e4bd76c5a81728524d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:26 UTC) from {username='slave'} at 127.0.0.1:38516
I20260620 21:49:26.152997 20810 tablet_replica.cc:330] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1: stopping tablet replica
I20260620 21:49:26.153024 20811 tablet_replica.cc:330] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382: stopping tablet replica
I20260620 21:49:26.153046 20442 tablet_service.cc:1514] Processing DeleteTablet for tablet 64ce5999c0a149e4bd76c5a81728524d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:26 UTC) from {username='slave'} at 127.0.0.1:38198
I20260620 21:49:26.153391 20811 raft_consensus.cc:2229] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:49:26.153441 20810 raft_consensus.cc:2229] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:49:26.153743 20811 raft_consensus.cc:2258] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:49:26.153901 20812 tablet_replica.cc:330] T 64ce5999c0a149e4bd76c5a81728524d P 7cd008e1efbb4b50bbb44c2b8ebe2bb3: stopping tablet replica
I20260620 21:49:26.154031 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 20380
I20260620 21:49:26.156085 20811 ts_tablet_manager.cc:1905] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:49:26.158164 20810 raft_consensus.cc:2258] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:49:26.159794 20811 ts_tablet_manager.cc:1918] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:49:26.159901 20811 log.cc:1196] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/64ce5999c0a149e4bd76c5a81728524d
I20260620 21:49:26.159904 20810 ts_tablet_manager.cc:1905] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:49:26.160210 20811 ts_tablet_manager.cc:1939] T 64ce5999c0a149e4bd76c5a81728524d P dd56f1c4f94c4245b3ce4996e2803382: Deleting consensus metadata
I20260620 21:49:26.161602 20305 catalog_manager.cc:4841] TS dd56f1c4f94c4245b3ce4996e2803382 (127.1.155.130:41659): tablet 64ce5999c0a149e4bd76c5a81728524d (table test-table [id=ec9038bee89948febda2a80b19de3501]) successfully deleted
W20260620 21:49:26.162108 20306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.155.129:34751 (error 108)
I20260620 21:49:26.162345 20810 ts_tablet_manager.cc:1918] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:49:26.162456 20810 log.cc:1196] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/64ce5999c0a149e4bd76c5a81728524d
I20260620 21:49:26.162729 20810 ts_tablet_manager.cc:1939] T 64ce5999c0a149e4bd76c5a81728524d P 055801db18b8425083011285766b96b1: Deleting consensus metadata
I20260620 21:49:26.163147 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:26.162842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:34751 (local address 127.0.0.1:38214)
0620 21:49:26.162988 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:26.163062 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:34751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 21:49:26.163208 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 20511
I20260620 21:49:26.164212 20304 catalog_manager.cc:4841] TS 055801db18b8425083011285766b96b1 (127.1.155.131:40323): tablet 64ce5999c0a149e4bd76c5a81728524d (table test-table [id=ec9038bee89948febda2a80b19de3501]) successfully deleted
W20260620 21:49:26.166330 20306 catalog_manager.cc:4584] TS 7cd008e1efbb4b50bbb44c2b8ebe2bb3 (127.1.155.129:34751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 64ce5999c0a149e4bd76c5a81728524d: Network error: Client connection negotiation failed: client connection to 127.1.155.129:34751: connect: Connection refused (error 111)
I20260620 21:49:26.168959 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 20642
I20260620 21:49:26.174549 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 20289
2026-06-20T21:49:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1003 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:26 dist-test-slave-5bvn krb5kdc[20816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:26 dist-test-slave-5bvn krb5kdc[20816](info): set up 2 sockets
Jun 20 21:49:26 dist-test-slave-5bvn krb5kdc[20816](info): commencing operation
krb5kdc: starting...
W20260620 21:49:27.233820 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:27 dist-test-slave-5bvn krb5kdc[20816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:49:27.361325 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:32945
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:32945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:27.468408 20829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:27.468710 20829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:27.468786 20829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:27.472788 20829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:27.472901 20829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:27.472944 20829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:27.472982 20829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:27.473013 20829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:27.478379 20829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:32945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:32945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:27.479470 20829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:27.480497 20829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:27.488997 20834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:27.489095 20829 server_base.cc:1034] running on GCE node
W20260620 21:49:27.489012 20835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:27.489012 20837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:27.490079 20829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:27.491227 20829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:27.492440 20829 hybrid_clock.cc:648] HybridClock initialized: now 1781992167492421 us; error 25 us; skew 500 ppm
Jun 20 21:49:27 dist-test-slave-5bvn krb5kdc[20816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:27.495429 20829 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:49:27.496368 20829 webserver.cc:460] Webserver started at http://127.1.155.190:40999/ using document root <none> and password file <none>
I20260620 21:49:27.496955 20829 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:27.497048 20829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:27.497284 20829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:27.499511 20829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "415e3f326d744db6b6e4dd16e8536bfb"
format_stamp: "Formatted at 2026-06-20 21:49:27 on dist-test-slave-5bvn"
server_key: "e2a36206300f791d2f25a1a571fefe95"
server_key_iv: "13b9fbfccebf4104fedb59d44285ff21"
server_key_version: "encryptionkey@0"
I20260620 21:49:27.500084 20829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "415e3f326d744db6b6e4dd16e8536bfb"
format_stamp: "Formatted at 2026-06-20 21:49:27 on dist-test-slave-5bvn"
server_key: "e2a36206300f791d2f25a1a571fefe95"
server_key_iv: "13b9fbfccebf4104fedb59d44285ff21"
server_key_version: "encryptionkey@0"
I20260620 21:49:27.504105 20829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:49:27.506943 20844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:27.508262 20829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 21:49:27.508416 20829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "415e3f326d744db6b6e4dd16e8536bfb"
format_stamp: "Formatted at 2026-06-20 21:49:27 on dist-test-slave-5bvn"
server_key: "e2a36206300f791d2f25a1a571fefe95"
server_key_iv: "13b9fbfccebf4104fedb59d44285ff21"
server_key_version: "encryptionkey@0"
I20260620 21:49:27.508548 20829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:27.527881 20829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:27.531294 20829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:27.531517 20829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:27.539261 20829 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:32945
I20260620 21:49:27.539289 20896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:32945 every 8 connection(s)
I20260620 21:49:27.540256 20829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:27.543015 20897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:27.547577 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 20829
I20260620 21:49:27.547729 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:27.548023 1646 external_mini_cluster.cc:1428] Setting key c889482c1a255337050f8b8f5bd4d4bf
I20260620 21:49:27.548642 20897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb: Bootstrap starting.
I20260620 21:49:27.551144 20897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:27.551877 20897 log.cc:824] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:27.554304 20897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb: No bootstrap required, opened a new log
Jun 20 21:49:27 dist-test-slave-5bvn krb5kdc[20816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:27.558543 20897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e3f326d744db6b6e4dd16e8536bfb" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 32945 } }
I20260620 21:49:27.559022 20897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:27.559098 20897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 415e3f326d744db6b6e4dd16e8536bfb, State: Initialized, Role: FOLLOWER
I20260620 21:49:27.559612 20897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e3f326d744db6b6e4dd16e8536bfb" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 32945 } }
I20260620 21:49:27.559788 20897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:27.559851 20897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:27.559968 20897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:27.561177 20897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e3f326d744db6b6e4dd16e8536bfb" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 32945 } }
I20260620 21:49:27.561543 20897 leader_election.cc:302] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 415e3f326d744db6b6e4dd16e8536bfb; no voters:
I20260620 21:49:27.561972 20897 leader_election.cc:288] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:27.562147 20902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:27.562455 20902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [term 1 LEADER]: Becoming Leader. State: Replica: 415e3f326d744db6b6e4dd16e8536bfb, State: Running, Role: LEADER
I20260620 21:49:27.562824 20902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e3f326d744db6b6e4dd16e8536bfb" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 32945 } }
I20260620 21:49:27.563262 20897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:27.563529 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:27.549483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.1.155.190:32945)
0620 21:49:27.549981 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:27.550001 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:27.550062 (+ 61us) server_negotiation.cc:407] Connection header received
0620 21:49:27.550690 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:27.550707 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:27.551021 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:27.551364 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:27.552216 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:27.553208 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:27.554220 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:27.554767 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:27.554958 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:27.557765 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:27.557773 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:27.557794 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:27.557825 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:27.559968 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:27.560477 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:27.560483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:27.560489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:27.560627 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:27.560937 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:27.560940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:27.560942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:27.561509 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:27.561700 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:27.562164 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 21:49:27.562493 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":139,"threads_started":1}
I20260620 21:49:27.564589 20904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "415e3f326d744db6b6e4dd16e8536bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e3f326d744db6b6e4dd16e8536bfb" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 32945 } } }
I20260620 21:49:27.564710 20904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:27.565062 20903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 415e3f326d744db6b6e4dd16e8536bfb. Latest consensus state: current_term: 1 leader_uuid: "415e3f326d744db6b6e4dd16e8536bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415e3f326d744db6b6e4dd16e8536bfb" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 32945 } } }
I20260620 21:49:27.565137 20903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb [sys.catalog]: This master's current role is: LEADER
W20260620 21:49:27.568311 20917 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:49:27.568405 20917 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:49:27.568506 20912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:27.569264 20912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:27.575846 20912 catalog_manager.cc:1347] Generated new cluster ID: fe35c3ade13e4933be5adfda36c9dab5
I20260620 21:49:27.575932 20912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:27.584048 20912 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:27.585114 20912 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:27.593170 20912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb: Generated new TSK 0
I20260620 21:49:27.594020 20912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.155.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
Entry for principal kudu/127.1.155.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab.
I20260620 21:49:27.630051 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:32945
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:27.741057 20923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:27.741315 20923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:27.741358 20923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:27.745417 20923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:27.745483 20923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:27.745546 20923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:49:27.750955 20923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.129.keytab
--principal=kudu/127.1.155.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:32945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:27.751981 20923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:27.752929 20923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:27.763268 20928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:27.763399 20923 server_base.cc:1034] running on GCE node
W20260620 21:49:27.763270 20929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:27.763589 20931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:27.764089 20923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:27.764991 20923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:27.766209 20923 hybrid_clock.cc:648] HybridClock initialized: now 1781992167766193 us; error 30 us; skew 500 ppm
Jun 20 21:49:27 dist-test-slave-5bvn krb5kdc[20816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:27.769164 20923 init.cc:377] Logged in from keytab as kudu/127.1.155.129@KRBTEST.COM (short username kudu)
I20260620 21:49:27.770153 20923 webserver.cc:460] Webserver started at http://127.1.155.129:34945/ using document root <none> and password file <none>
I20260620 21:49:27.770722 20923 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:27.770817 20923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:27.771049 20923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:27.773108 20923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "993a0921e3bf4cb49ccdf82504ebb317"
format_stamp: "Formatted at 2026-06-20 21:49:27 on dist-test-slave-5bvn"
server_key: "9d78f4f75f69301cedaf26658b82f9de"
server_key_iv: "8e7c2d85d002587a122cc8b4b3e44f3a"
server_key_version: "encryptionkey@0"
I20260620 21:49:27.773694 20923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "993a0921e3bf4cb49ccdf82504ebb317"
format_stamp: "Formatted at 2026-06-20 21:49:27 on dist-test-slave-5bvn"
server_key: "9d78f4f75f69301cedaf26658b82f9de"
server_key_iv: "8e7c2d85d002587a122cc8b4b3e44f3a"
server_key_version: "encryptionkey@0"
I20260620 21:49:27.777510 20923 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:49:27.779985 20938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:27.781247 20923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:49:27.781445 20923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "993a0921e3bf4cb49ccdf82504ebb317"
format_stamp: "Formatted at 2026-06-20 21:49:27 on dist-test-slave-5bvn"
server_key: "9d78f4f75f69301cedaf26658b82f9de"
server_key_iv: "8e7c2d85d002587a122cc8b4b3e44f3a"
server_key_version: "encryptionkey@0"
I20260620 21:49:27.781589 20923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:27.820534 20923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:27.824362 20923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:27.824640 20923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:27.825335 20923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:27.826299 20923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:27.826397 20923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:27.826484 20923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:27.826527 20923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:27.837218 20923 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:37187
I20260620 21:49:27.837241 21051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:37187 every 8 connection(s)
I20260620 21:49:27.838261 20923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:49:27.847083 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 20923
I20260620 21:49:27.847200 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:49:27.847651 1646 external_mini_cluster.cc:1428] Setting key b752dedd75431a36c7850c4fa1a8d3f4
Jun 20 21:49:27 dist-test-slave-5bvn krb5kdc[20816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.129@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:27.853401 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:27.840282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:44363 (local address 127.1.155.190:32945)
0620 21:49:27.840475 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:27.840481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:27.841352 (+ 871us) server_negotiation.cc:407] Connection header received
0620 21:49:27.842362 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:27.842365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:27.842413 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:27.842501 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:27.843753 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:27.844297 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:27.845588 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:27.845806 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:27.845909 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:27.848826 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:27.848830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:27.848832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:27.848857 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:27.850748 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:27.851455 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:27.851462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:27.851466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:27.851573 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:27.852066 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:27.852071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:27.852072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:27.852309 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:27.852394 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:27.853103 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 21:49:27.853231 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 21:49:27.854377 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:27.840660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:32945 (local address 127.1.155.129:44363)
0620 21:49:27.841163 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:27.841211 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:27.842129 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:27.842673 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:27.842680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:27.843058 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:27.843588 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:27.843603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:27.844456 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:27.844459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:27.845473 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:27.845483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:27.846020 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:27.846024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:27.846167 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:27.846829 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:27.846845 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:27.848654 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:27.850885 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:27.850890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:27.850906 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:27.851284 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:27.851704 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:27.851708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:27.851710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:27.851920 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:27.852481 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:27.852488 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:27.852815 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 21:49:27.853048 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 21:49:27.853381 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":125,"threads_started":1}
I20260620 21:49:27.855728 21052 heartbeater.cc:347] Connected to a master server at 127.1.155.190:32945
I20260620 21:49:27.855983 21052 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:27.856500 21052 heartbeater.cc:511] Master 127.1.155.190:32945 requested a full tablet report, sending...
I20260620 21:49:27.858103 20861 ts_manager.cc:194] Registered new tserver with Master: 993a0921e3bf4cb49ccdf82504ebb317 (127.1.155.129:37187)
WARNING: no policy specified for kudu/127.1.155.130@KRBTEST.COM; defaulting to no policy
I20260620 21:49:27.859242 20861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.129@KRBTEST.COM'} at 127.1.155.129:44363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
Entry for principal kudu/127.1.155.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab.
I20260620 21:49:27.877465 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:32945
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:27.987485 21057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:27.987752 21057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:27.987797 21057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:27.991838 21057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:27.991901 21057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:27.991981 21057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:49:27.997296 21057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.130.keytab
--principal=kudu/127.1.155.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:32945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:27.998296 21057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:27.999238 21057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:28.008685 21065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:28.008736 21062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:28.008805 21063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:28.009039 21057 server_base.cc:1034] running on GCE node
I20260620 21:49:28.009423 21057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:28.010121 21057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:28.011333 21057 hybrid_clock.cc:648] HybridClock initialized: now 1781992168011319 us; error 25 us; skew 500 ppm
Jun 20 21:49:28 dist-test-slave-5bvn krb5kdc[20816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992168, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:28.014076 21057 init.cc:377] Logged in from keytab as kudu/127.1.155.130@KRBTEST.COM (short username kudu)
I20260620 21:49:28.014958 21057 webserver.cc:460] Webserver started at http://127.1.155.130:32869/ using document root <none> and password file <none>
I20260620 21:49:28.015522 21057 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:28.015614 21057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:28.015841 21057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:28.017920 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "b00494a98cd843b6ac718a43afa87ebf"
format_stamp: "Formatted at 2026-06-20 21:49:28 on dist-test-slave-5bvn"
server_key: "caaf20b9b683a53b9728f0ed847ff11c"
server_key_iv: "574de7f37e0f20291a2be0de7154c37e"
server_key_version: "encryptionkey@0"
I20260620 21:49:28.018512 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "b00494a98cd843b6ac718a43afa87ebf"
format_stamp: "Formatted at 2026-06-20 21:49:28 on dist-test-slave-5bvn"
server_key: "caaf20b9b683a53b9728f0ed847ff11c"
server_key_iv: "574de7f37e0f20291a2be0de7154c37e"
server_key_version: "encryptionkey@0"
I20260620 21:49:28.022300 21057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:49:28.024811 21072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:28.026242 21057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:49:28.026387 21057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "b00494a98cd843b6ac718a43afa87ebf"
format_stamp: "Formatted at 2026-06-20 21:49:28 on dist-test-slave-5bvn"
server_key: "caaf20b9b683a53b9728f0ed847ff11c"
server_key_iv: "574de7f37e0f20291a2be0de7154c37e"
server_key_version: "encryptionkey@0"
I20260620 21:49:28.026516 21057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:28.040310 21057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:28.043668 21057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:28.043895 21057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:28.044505 21057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:28.045338 21057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:28.045411 21057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:28.045477 21057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:28.045528 21057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:28.054849 21057 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:41549
I20260620 21:49:28.054875 21185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:41549 every 8 connection(s)
I20260620 21:49:28.055713 21057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:49:28.064126 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 21057
I20260620 21:49:28.064276 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:49:28.064579 1646 external_mini_cluster.cc:1428] Setting key e0850a939ca98f11bd02dac7ae55db36
Jun 20 21:49:28 dist-test-slave-5bvn krb5kdc[20816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992168, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.130@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:28.069931 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.057582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:60299 (local address 127.1.155.190:32945)
0620 21:49:28.057753 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:28.057757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:28.058828 (+ 1071us) server_negotiation.cc:407] Connection header received
0620 21:49:28.059675 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:28.059678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:28.059725 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:28.059803 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:28.060915 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.061459 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.062689 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.062920 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.063038 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.066045 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:28.066048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:28.066051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:28.066076 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:28.067538 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:28.068234 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:28.068237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:28.068238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:28.068320 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:28.068737 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:28.068739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:28.068741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:28.068921 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:28.069030 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:28.069613 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 21:49:28.069729 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 21:49:28.070837 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.058024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:32945 (local address 127.1.155.130:60299)
0620 21:49:28.058680 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:28.058720 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:28.059477 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:28.059956 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:28.059962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:28.060322 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:28.060780 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.060793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.061569 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.061572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.062533 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.062565 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.063100 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.063105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.063343 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.063993 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:28.064009 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:28.065903 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:28.067676 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:28.067682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:28.067697 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:28.068086 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:28.068419 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:28.068423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:28.068427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:28.068616 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:28.069027 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:28.069033 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:28.069288 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 21:49:28.069448 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 21:49:28.069707 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":111,"threads_started":1}
I20260620 21:49:28.072299 21186 heartbeater.cc:347] Connected to a master server at 127.1.155.190:32945
I20260620 21:49:28.072597 21186 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:28.073137 21186 heartbeater.cc:511] Master 127.1.155.190:32945 requested a full tablet report, sending...
I20260620 21:49:28.074306 20861 ts_manager.cc:194] Registered new tserver with Master: b00494a98cd843b6ac718a43afa87ebf (127.1.155.130:41549)
I20260620 21:49:28.075142 20861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.130@KRBTEST.COM'} at 127.1.155.130:60299
WARNING: no policy specified for kudu/127.1.155.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
Entry for principal kudu/127.1.155.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab.
I20260620 21:49:28.094970 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:32945
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:28.207643 21191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:28.207945 21191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:28.208019 21191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:28.212287 21191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:28.212370 21191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:28.212460 21191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:49:28.218559 21191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.131.keytab
--principal=kudu/127.1.155.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:32945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:28.219643 21191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:28.220702 21191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:28.230430 21197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:28.230434 21196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:28.230448 21199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:28.230930 21191 server_base.cc:1034] running on GCE node
I20260620 21:49:28.231412 21191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:28.232116 21191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:28.233347 21191 hybrid_clock.cc:648] HybridClock initialized: now 1781992168233312 us; error 48 us; skew 500 ppm
Jun 20 21:49:28 dist-test-slave-5bvn krb5kdc[20816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992168, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:28.236370 21191 init.cc:377] Logged in from keytab as kudu/127.1.155.131@KRBTEST.COM (short username kudu)
I20260620 21:49:28.237264 21191 webserver.cc:460] Webserver started at http://127.1.155.131:35481/ using document root <none> and password file <none>
I20260620 21:49:28.237900 21191 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:28.238045 21191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:28.238287 21191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:28.240331 21191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "573c36fc79ef47519244b23e8dbe6b41"
format_stamp: "Formatted at 2026-06-20 21:49:28 on dist-test-slave-5bvn"
server_key: "52992a8f0b06ea65275e22094d6d962e"
server_key_iv: "555932046b46b9116439a999ef8450cb"
server_key_version: "encryptionkey@0"
I20260620 21:49:28.240890 21191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "573c36fc79ef47519244b23e8dbe6b41"
format_stamp: "Formatted at 2026-06-20 21:49:28 on dist-test-slave-5bvn"
server_key: "52992a8f0b06ea65275e22094d6d962e"
server_key_iv: "555932046b46b9116439a999ef8450cb"
server_key_version: "encryptionkey@0"
I20260620 21:49:28.244649 21191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:49:28.247143 21206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:28.248301 21191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:49:28.248447 21191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "573c36fc79ef47519244b23e8dbe6b41"
format_stamp: "Formatted at 2026-06-20 21:49:28 on dist-test-slave-5bvn"
server_key: "52992a8f0b06ea65275e22094d6d962e"
server_key_iv: "555932046b46b9116439a999ef8450cb"
server_key_version: "encryptionkey@0"
I20260620 21:49:28.248566 21191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:28.270501 21191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:28.273677 21191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:28.273857 21191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:28.274369 21191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:28.275118 21191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:28.275167 21191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:28.275218 21191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:28.275233 21191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:28.283989 21191 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:34261
I20260620 21:49:28.284039 21319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:34261 every 8 connection(s)
I20260620 21:49:28.284909 21191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:49:28.291719 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 21191
I20260620 21:49:28.291898 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:49:28.292223 1646 external_mini_cluster.cc:1428] Setting key 78b300a5212cc04f0d7408236747bc04
Jun 20 21:49:28 dist-test-slave-5bvn krb5kdc[20816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992168, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.131@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:28.299476 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.286900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:56243 (local address 127.1.155.190:32945)
0620 21:49:28.287081 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:28.287085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:28.288166 (+ 1081us) server_negotiation.cc:407] Connection header received
0620 21:49:28.289048 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:28.289050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:28.289095 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:28.289183 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:28.290574 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.291127 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.292409 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.292691 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.292820 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.295934 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:28.295944 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:28.295946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:28.295970 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:28.297469 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:28.298065 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:28.298068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:28.298069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:28.298160 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:28.298523 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:28.298525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:28.298526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:28.298702 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:28.298778 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:28.299262 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 21:49:28.299359 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 21:49:28.300263 21322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.287317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:32945 (local address 127.1.155.131:56243)
0620 21:49:28.288018 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:28.288061 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:28.288852 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:28.289385 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:28.289392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:28.289745 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:28.290434 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.290449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.291263 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.291265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.292256 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.292266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.292876 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.292879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.293095 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.293946 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:28.293966 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:28.295732 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:28.297585 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:28.297590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:28.297602 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:28.297945 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:28.298278 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:28.298281 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:28.298282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:28.298420 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:28.298791 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:28.298796 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:28.299032 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 21:49:28.299185 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 21:49:28.299430 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":120,"threads_started":1}
I20260620 21:49:28.301412 21320 heartbeater.cc:347] Connected to a master server at 127.1.155.190:32945
I20260620 21:49:28.301687 21320 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:28.302289 21320 heartbeater.cc:511] Master 127.1.155.190:32945 requested a full tablet report, sending...
I20260620 21:49:28.303314 20861 ts_manager.cc:194] Registered new tserver with Master: 573c36fc79ef47519244b23e8dbe6b41 (127.1.155.131:34261)
I20260620 21:49:28.303925 20861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.155.131@KRBTEST.COM'} at 127.1.155.131:56243
I20260620 21:49:28.306318 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:49:28.315742 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.308187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46438 (local address 127.1.155.190:32945)
0620 21:49:28.308301 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:28.308305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:28.308495 (+ 190us) server_negotiation.cc:407] Connection header received
0620 21:49:28.308598 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:28.308600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:28.308635 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:28.308701 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:28.309614 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.310253 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.311066 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.311306 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.311405 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.312822 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:28.312825 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:28.312827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:28.312851 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:28.314341 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:28.314712 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:28.314714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:28.314716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:28.314788 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:28.315021 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:28.315023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:28.315024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:28.315220 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:28.315322 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:28.315538 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 21:49:28.315631 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:49:28.318593 20861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:49:28.321533 20861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:49:28.336983 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.331076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.1.155.129:37187)
0620 21:49:28.331933 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:28.331950 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:28.331985 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:49:28.332061 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:28.332066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:28.332245 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:28.332392 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:28.332830 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.333898 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.335294 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.336165 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.336291 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.336453 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:28.336706 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 21:49:28.336806 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":91,"threads_started":1}
I20260620 21:49:28.337019 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.330339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:41549 (local address 127.0.0.1:41478)
0620 21:49:28.331335 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:28.331351 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:28.331435 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:28.332026 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:28.332031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:28.332054 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:28.332248 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.332256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.333605 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.333609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.335302 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.335313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.336330 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.336334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.336491 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.336545 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 21:49:28.336652 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 21:49:28.336828 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":254,"thread_start_us":119,"threads_started":1}
I20260620 21:49:28.337164 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.331123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.1.155.130:41549)
0620 21:49:28.331517 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:28.331528 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:28.331551 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:49:28.331626 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:28.331631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:28.331824 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:28.332026 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:28.332366 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.333415 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.335443 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.336201 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.336740 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.336867 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:28.336912 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 21:49:28.337005 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":107,"threads_started":1}
I20260620 21:49:28.337231 21334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.330153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:34261 (local address 127.0.0.1:49028)
0620 21:49:28.331032 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:28.331081 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:28.331221 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:28.332062 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:28.332065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:28.332086 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:28.332249 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.332258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.333510 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.333517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.335439 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.335447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.336306 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.336310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.336938 (+ 628us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.336969 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 21:49:28.337049 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 21:49:28.337118 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":856,"thread_start_us":95,"threads_started":1}
I20260620 21:49:28.337682 21332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.330273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.1.155.131:34261)
0620 21:49:28.330751 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:28.330764 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:28.331383 (+ 619us) server_negotiation.cc:407] Connection header received
0620 21:49:28.331486 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:28.331493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:28.331730 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:28.331888 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:28.332402 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.333353 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.335569 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.336168 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.336309 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.336474 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:28.337400 (+ 926us) server_negotiation.cc:299] Negotiation successful
0620 21:49:28.337520 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":125,"threads_started":1}
I20260620 21:49:28.337707 21331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.330692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:37187 (local address 127.0.0.1:43958)
0620 21:49:28.331600 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:28.331614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:28.331681 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:28.332573 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:28.332576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:28.332595 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:28.332682 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.332688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.334086 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.334090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.335166 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.335176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.336452 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.336456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.336554 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.336580 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 21:49:28.336646 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 21:49:28.336828 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":8,"thread_start_us":108,"threads_started":1}
I20260620 21:49:28.340610 20986 tablet_service.cc:1467] Processing CreateTablet for tablet b968941e5ca2494e8a1786fb303b3a81 (DEFAULT_TABLE table=test-table [id=6eb7c567c240418e835218a462c7d981]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:28.340695 21254 tablet_service.cc:1467] Processing CreateTablet for tablet b968941e5ca2494e8a1786fb303b3a81 (DEFAULT_TABLE table=test-table [id=6eb7c567c240418e835218a462c7d981]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:28.340694 21120 tablet_service.cc:1467] Processing CreateTablet for tablet b968941e5ca2494e8a1786fb303b3a81 (DEFAULT_TABLE table=test-table [id=6eb7c567c240418e835218a462c7d981]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:28.341693 20986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b968941e5ca2494e8a1786fb303b3a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:28.341779 21120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b968941e5ca2494e8a1786fb303b3a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:28.341785 21254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b968941e5ca2494e8a1786fb303b3a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:28.348436 21337 tablet_bootstrap.cc:492] T b968941e5ca2494e8a1786fb303b3a81 P 993a0921e3bf4cb49ccdf82504ebb317: Bootstrap starting.
I20260620 21:49:28.350227 21338 tablet_bootstrap.cc:492] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41: Bootstrap starting.
I20260620 21:49:28.351437 21337 tablet_bootstrap.cc:654] T b968941e5ca2494e8a1786fb303b3a81 P 993a0921e3bf4cb49ccdf82504ebb317: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:28.351867 21339 tablet_bootstrap.cc:492] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf: Bootstrap starting.
I20260620 21:49:28.352841 21337 log.cc:824] T b968941e5ca2494e8a1786fb303b3a81 P 993a0921e3bf4cb49ccdf82504ebb317: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:28.352972 21338 tablet_bootstrap.cc:654] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:28.353827 21338 log.cc:824] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:28.354437 21339 tablet_bootstrap.cc:654] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:28.355239 21339 log.cc:824] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:28.355306 21337 tablet_bootstrap.cc:492] T b968941e5ca2494e8a1786fb303b3a81 P 993a0921e3bf4cb49ccdf82504ebb317: No bootstrap required, opened a new log
I20260620 21:49:28.355490 21337 ts_tablet_manager.cc:1397] T b968941e5ca2494e8a1786fb303b3a81 P 993a0921e3bf4cb49ccdf82504ebb317: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 21:49:28.355916 21338 tablet_bootstrap.cc:492] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41: No bootstrap required, opened a new log
I20260620 21:49:28.356125 21338 ts_tablet_manager.cc:1397] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:49:28.358649 21337 raft_consensus.cc:348] T b968941e5ca2494e8a1786fb303b3a81 P 993a0921e3bf4cb49ccdf82504ebb317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c36fc79ef47519244b23e8dbe6b41" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34261 } } peers { permanent_uuid: "b00494a98cd843b6ac718a43afa87ebf" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41549 } } peers { permanent_uuid: "993a0921e3bf4cb49ccdf82504ebb317" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 37187 } }
I20260620 21:49:28.358916 21337 raft_consensus.cc:374] T b968941e5ca2494e8a1786fb303b3a81 P 993a0921e3bf4cb49ccdf82504ebb317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:28.358963 21337 raft_consensus.cc:729] T b968941e5ca2494e8a1786fb303b3a81 P 993a0921e3bf4cb49ccdf82504ebb317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993a0921e3bf4cb49ccdf82504ebb317, State: Initialized, Role: FOLLOWER
I20260620 21:49:28.359395 21338 raft_consensus.cc:348] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c36fc79ef47519244b23e8dbe6b41" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34261 } } peers { permanent_uuid: "b00494a98cd843b6ac718a43afa87ebf" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41549 } } peers { permanent_uuid: "993a0921e3bf4cb49ccdf82504ebb317" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 37187 } }
I20260620 21:49:28.359480 21337 consensus_queue.cc:260] T b968941e5ca2494e8a1786fb303b3a81 P 993a0921e3bf4cb49ccdf82504ebb317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c36fc79ef47519244b23e8dbe6b41" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34261 } } peers { permanent_uuid: "b00494a98cd843b6ac718a43afa87ebf" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41549 } } peers { permanent_uuid: "993a0921e3bf4cb49ccdf82504ebb317" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 37187 } }
I20260620 21:49:28.359640 21338 raft_consensus.cc:374] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:28.359706 21338 raft_consensus.cc:729] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573c36fc79ef47519244b23e8dbe6b41, State: Initialized, Role: FOLLOWER
I20260620 21:49:28.360154 21338 consensus_queue.cc:260] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c36fc79ef47519244b23e8dbe6b41" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34261 } } peers { permanent_uuid: "b00494a98cd843b6ac718a43afa87ebf" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41549 } } peers { permanent_uuid: "993a0921e3bf4cb49ccdf82504ebb317" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 37187 } }
I20260620 21:49:28.360337 21052 heartbeater.cc:503] Master 127.1.155.190:32945 was elected leader, sending a full tablet report...
I20260620 21:49:28.360481 21337 ts_tablet_manager.cc:1428] T b968941e5ca2494e8a1786fb303b3a81 P 993a0921e3bf4cb49ccdf82504ebb317: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 21:49:28.360632 21339 tablet_bootstrap.cc:492] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf: No bootstrap required, opened a new log
I20260620 21:49:28.360836 21339 ts_tablet_manager.cc:1397] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260620 21:49:28.361029 21320 heartbeater.cc:503] Master 127.1.155.190:32945 was elected leader, sending a full tablet report...
I20260620 21:49:28.361716 21338 ts_tablet_manager.cc:1428] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:49:28.364781 21339 raft_consensus.cc:348] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c36fc79ef47519244b23e8dbe6b41" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34261 } } peers { permanent_uuid: "b00494a98cd843b6ac718a43afa87ebf" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41549 } } peers { permanent_uuid: "993a0921e3bf4cb49ccdf82504ebb317" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 37187 } }
I20260620 21:49:28.365108 21339 raft_consensus.cc:374] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:28.365216 21339 raft_consensus.cc:729] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00494a98cd843b6ac718a43afa87ebf, State: Initialized, Role: FOLLOWER
I20260620 21:49:28.365654 21339 consensus_queue.cc:260] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c36fc79ef47519244b23e8dbe6b41" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34261 } } peers { permanent_uuid: "b00494a98cd843b6ac718a43afa87ebf" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41549 } } peers { permanent_uuid: "993a0921e3bf4cb49ccdf82504ebb317" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 37187 } }
I20260620 21:49:28.366289 21186 heartbeater.cc:503] Master 127.1.155.190:32945 was elected leader, sending a full tablet report...
I20260620 21:49:28.366559 21339 ts_tablet_manager.cc:1428] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:49:28.386193 21344 raft_consensus.cc:482] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:49:28.386458 21344 raft_consensus.cc:504] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c36fc79ef47519244b23e8dbe6b41" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34261 } } peers { permanent_uuid: "b00494a98cd843b6ac718a43afa87ebf" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41549 } } peers { permanent_uuid: "993a0921e3bf4cb49ccdf82504ebb317" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 37187 } }
I20260620 21:49:28.387656 21344 leader_election.cc:288] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00494a98cd843b6ac718a43afa87ebf (127.1.155.130:41549), 993a0921e3bf4cb49ccdf82504ebb317 (127.1.155.129:37187)
I20260620 21:49:28.392400 21322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.387825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:41549 (local address 127.1.155.131:51647)
0620 21:49:28.387938 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:28.387955 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:28.388061 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:28.388365 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:28.388369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:28.388398 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:28.388537 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.388544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.389604 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.389607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.390950 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.390961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.391914 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.391920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.392035 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.392062 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:49:28.392148 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 21:49:28.392257 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":62}
I20260620 21:49:28.392412 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.388105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:34783 (local address 127.1.155.129:37187)
0620 21:49:28.388224 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:28.388228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:28.388474 (+ 246us) server_negotiation.cc:407] Connection header received
0620 21:49:28.388572 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:28.388574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:28.388651 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:28.388734 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:28.388970 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.389694 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.391151 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.391776 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.391959 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.392018 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:28.392198 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 21:49:28.392274 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 21:49:28.392417 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.387971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:51647 (local address 127.1.155.130:41549)
0620 21:49:28.388107 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:28.388113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:28.388131 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:49:28.388186 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:28.388189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:28.388253 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:28.388324 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:28.388671 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.389471 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.391151 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.391776 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.391920 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.391979 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:28.392198 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 21:49:28.392274 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:49:28.392674 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.387982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:37187 (local address 127.1.155.131:34783)
0620 21:49:28.388362 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:28.388376 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:28.388473 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:28.388754 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:28.388756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:28.388776 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:28.388855 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.388859 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.389796 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.389799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.390987 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.390994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.391942 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.391945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.392068 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.392088 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 21:49:28.392157 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 21:49:28.392518 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":295,"thread_start_us":113,"threads_started":1}
I20260620 21:49:28.393065 21140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b968941e5ca2494e8a1786fb303b3a81" candidate_uuid: "573c36fc79ef47519244b23e8dbe6b41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00494a98cd843b6ac718a43afa87ebf" is_pre_election: true
I20260620 21:49:28.393281 21006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b968941e5ca2494e8a1786fb303b3a81" candidate_uuid: "573c36fc79ef47519244b23e8dbe6b41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993a0921e3bf4cb49ccdf82504ebb317" is_pre_election: true
I20260620 21:49:28.393411 21140 raft_consensus.cc:2454] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573c36fc79ef47519244b23e8dbe6b41 in term 0.
I20260620 21:49:28.393573 21006 raft_consensus.cc:2454] T b968941e5ca2494e8a1786fb303b3a81 P 993a0921e3bf4cb49ccdf82504ebb317 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573c36fc79ef47519244b23e8dbe6b41 in term 0.
I20260620 21:49:28.394032 21208 leader_election.cc:302] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 573c36fc79ef47519244b23e8dbe6b41, 993a0921e3bf4cb49ccdf82504ebb317, b00494a98cd843b6ac718a43afa87ebf; no voters:
I20260620 21:49:28.394323 21344 raft_consensus.cc:2781] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:49:28.394462 21344 raft_consensus.cc:482] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:49:28.394532 21344 raft_consensus.cc:3037] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:28.395680 21344 raft_consensus.cc:504] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c36fc79ef47519244b23e8dbe6b41" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34261 } } peers { permanent_uuid: "b00494a98cd843b6ac718a43afa87ebf" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41549 } } peers { permanent_uuid: "993a0921e3bf4cb49ccdf82504ebb317" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 37187 } }
I20260620 21:49:28.396107 21344 leader_election.cc:288] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [CANDIDATE]: Term 1 election: Requested vote from peers b00494a98cd843b6ac718a43afa87ebf (127.1.155.130:41549), 993a0921e3bf4cb49ccdf82504ebb317 (127.1.155.129:37187)
I20260620 21:49:28.396466 21140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b968941e5ca2494e8a1786fb303b3a81" candidate_uuid: "573c36fc79ef47519244b23e8dbe6b41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00494a98cd843b6ac718a43afa87ebf"
I20260620 21:49:28.396515 21006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b968941e5ca2494e8a1786fb303b3a81" candidate_uuid: "573c36fc79ef47519244b23e8dbe6b41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993a0921e3bf4cb49ccdf82504ebb317"
I20260620 21:49:28.396610 21140 raft_consensus.cc:3037] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:28.396637 21006 raft_consensus.cc:3037] T b968941e5ca2494e8a1786fb303b3a81 P 993a0921e3bf4cb49ccdf82504ebb317 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:28.397699 21140 raft_consensus.cc:2454] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573c36fc79ef47519244b23e8dbe6b41 in term 1.
I20260620 21:49:28.397709 21006 raft_consensus.cc:2454] T b968941e5ca2494e8a1786fb303b3a81 P 993a0921e3bf4cb49ccdf82504ebb317 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573c36fc79ef47519244b23e8dbe6b41 in term 1.
I20260620 21:49:28.398120 21207 leader_election.cc:302] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573c36fc79ef47519244b23e8dbe6b41, 993a0921e3bf4cb49ccdf82504ebb317; no voters:
I20260620 21:49:28.398378 21344 raft_consensus.cc:2781] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:28.398634 21344 raft_consensus.cc:686] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [term 1 LEADER]: Becoming Leader. State: Replica: 573c36fc79ef47519244b23e8dbe6b41, State: Running, Role: LEADER
I20260620 21:49:28.398998 21344 consensus_queue.cc:237] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c36fc79ef47519244b23e8dbe6b41" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34261 } } peers { permanent_uuid: "b00494a98cd843b6ac718a43afa87ebf" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41549 } } peers { permanent_uuid: "993a0921e3bf4cb49ccdf82504ebb317" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 37187 } }
I20260620 21:49:28.402212 20861 catalog_manager.cc:5495] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 573c36fc79ef47519244b23e8dbe6b41 (127.1.155.131). New cstate: current_term: 1 leader_uuid: "573c36fc79ef47519244b23e8dbe6b41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c36fc79ef47519244b23e8dbe6b41" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b00494a98cd843b6ac718a43afa87ebf" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "993a0921e3bf4cb49ccdf82504ebb317" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 37187 } health_report { overall_health: UNKNOWN } } }
I20260620 21:49:28.416215 21332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.412379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49042 (local address 127.1.155.131:34261)
0620 21:49:28.412525 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:28.412529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:28.412544 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:49:28.412687 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:28.412689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:28.412741 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:28.412851 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:49:28.413127 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.413627 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.414578 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.414803 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.414918 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.415352 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:49:28.415796 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:49:28.415942 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:28.415983 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 21:49:28.416041 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:49:28.422224 21006 raft_consensus.cc:1261] T b968941e5ca2494e8a1786fb303b3a81 P 993a0921e3bf4cb49ccdf82504ebb317 [term 1 FOLLOWER]: Refusing update from remote peer 573c36fc79ef47519244b23e8dbe6b41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:49:28.422224 21140 raft_consensus.cc:1261] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf [term 1 FOLLOWER]: Refusing update from remote peer 573c36fc79ef47519244b23e8dbe6b41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:49:28.422971 21344 consensus_queue.cc:1035] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00494a98cd843b6ac718a43afa87ebf" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 41549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:49:28.423156 21347 consensus_queue.cc:1035] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "993a0921e3bf4cb49ccdf82504ebb317" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 37187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:49:28.431267 21350 mvcc.cc:205] Tried to move back new op lower bound from 7299039921849753600 to 7299039921764929536. Current Snapshot: MvccSnapshot[applied={T|T < 7299039921849753600}]
I20260620 21:49:28.431571 21352 mvcc.cc:205] Tried to move back new op lower bound from 7299039921849753600 to 7299039921764929536. Current Snapshot: MvccSnapshot[applied={T|T < 7299039921849753600}]
I20260620 21:49:28.441699 20861 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:49:28.441910 20861 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:49:28.444403 20861 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 415e3f326d744db6b6e4dd16e8536bfb: Sending DeleteTablet for 3 replicas of tablet b968941e5ca2494e8a1786fb303b3a81
I20260620 21:49:28.445207 21254 tablet_service.cc:1514] Processing DeleteTablet for tablet b968941e5ca2494e8a1786fb303b3a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:28 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:49028
I20260620 21:49:28.445226 21120 tablet_service.cc:1514] Processing DeleteTablet for tablet b968941e5ca2494e8a1786fb303b3a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:28 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:41478
I20260620 21:49:28.445219 20986 tablet_service.cc:1514] Processing DeleteTablet for tablet b968941e5ca2494e8a1786fb303b3a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:28 UTC) from {username='kudu', principal='kudu/127.1.155.190@KRBTEST.COM'} at 127.0.0.1:43958
I20260620 21:49:28.445914 21360 tablet_replica.cc:330] T b968941e5ca2494e8a1786fb303b3a81 P 993a0921e3bf4cb49ccdf82504ebb317: stopping tablet replica
I20260620 21:49:28.446332 21361 tablet_replica.cc:330] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41: stopping tablet replica
I20260620 21:49:28.446624 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 20923
I20260620 21:49:28.447417 21361 raft_consensus.cc:2229] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:49:28.447793 21361 raft_consensus.cc:2258] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:49:28.449162 21361 ts_tablet_manager.cc:1905] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:49:28.451345 21361 ts_tablet_manager.cc:1918] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:49:28.451462 21361 log.cc:1196] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/b968941e5ca2494e8a1786fb303b3a81
I20260620 21:49:28.451823 21361 ts_tablet_manager.cc:1939] T b968941e5ca2494e8a1786fb303b3a81 P 573c36fc79ef47519244b23e8dbe6b41: Deleting consensus metadata
I20260620 21:49:28.452924 21359 tablet_replica.cc:330] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf: stopping tablet replica
I20260620 21:49:28.452915 20848 catalog_manager.cc:4841] TS 573c36fc79ef47519244b23e8dbe6b41 (127.1.155.131:34261): tablet b968941e5ca2494e8a1786fb303b3a81 (table test-table [id=6eb7c567c240418e835218a462c7d981]) successfully deleted
I20260620 21:49:28.453284 21359 raft_consensus.cc:2229] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:49:28.453619 21359 raft_consensus.cc:2258] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 21:49:28.454627 20845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.155.129:37187 (error 108)
I20260620 21:49:28.455173 21359 ts_tablet_manager.cc:1905] T b968941e5ca2494e8a1786fb303b3a81 P b00494a98cd843b6ac718a43afa87ebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:49:28.455413 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 21057
I20260620 21:49:28.456995 21331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.456690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:37187 (local address 127.0.0.1:43972)
0620 21:49:28.456823 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:28.456883 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 21:49:28.457222 20845 catalog_manager.cc:4584] TS 993a0921e3bf4cb49ccdf82504ebb317 (127.1.155.129:37187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b968941e5ca2494e8a1786fb303b3a81: Network error: Client connection negotiation failed: client connection to 127.1.155.129:37187: connect: Connection refused (error 111)
I20260620 21:49:28.460947 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 21191
I20260620 21:49:28.461064 21331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.460770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:41549 (local address 127.0.0.1:41492)
0620 21:49:28.460915 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:28.460976 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.130:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 21:49:28.466598 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 20829
2026-06-20T21:49:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2293 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T21:49:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:28Z Disabled control of system clock
I20260620 21:49:28.497575 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:43671
--webserver_interface=127.1.155.190
--webserver_port=0
--builtin_ntp_servers=127.1.155.148:37823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:43671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:49:28.609882 21366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:28.610186 21366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:28.610229 21366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:28.614450 21366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:28.614526 21366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:28.614544 21366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:28.614565 21366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:28.614584 21366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:28.619441 21366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:37823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:43671
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:43671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:28.620486 21366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:28.621901 21366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:28.630488 21371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:28.630575 21374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:28.630488 21372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:28.630905 21366 server_base.cc:1034] running on GCE node
I20260620 21:49:28.631477 21366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:28.632696 21366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:28.633982 21366 hybrid_clock.cc:648] HybridClock initialized: now 1781992168633957 us; error 39 us; skew 500 ppm
I20260620 21:49:28.636092 21366 webserver.cc:460] Webserver started at http://127.1.155.190:43423/ using document root <none> and password file <none>
I20260620 21:49:28.636734 21366 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:28.636840 21366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:28.637068 21366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:28.639314 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "7af78a8de91e48bca5e84c4dfadb1aff"
format_stamp: "Formatted at 2026-06-20 21:49:28 on dist-test-slave-5bvn"
server_key: "d5e9a5f24d5482cc94f5845867d23445"
server_key_iv: "6a422e6871f86371a3adc97280114791"
server_key_version: "encryptionkey@0"
I20260620 21:49:28.639909 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "7af78a8de91e48bca5e84c4dfadb1aff"
format_stamp: "Formatted at 2026-06-20 21:49:28 on dist-test-slave-5bvn"
server_key: "d5e9a5f24d5482cc94f5845867d23445"
server_key_iv: "6a422e6871f86371a3adc97280114791"
server_key_version: "encryptionkey@0"
I20260620 21:49:28.643777 21366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 21:49:28.646373 21380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:28.647794 21366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:49:28.647977 21366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "7af78a8de91e48bca5e84c4dfadb1aff"
format_stamp: "Formatted at 2026-06-20 21:49:28 on dist-test-slave-5bvn"
server_key: "d5e9a5f24d5482cc94f5845867d23445"
server_key_iv: "6a422e6871f86371a3adc97280114791"
server_key_version: "encryptionkey@0"
I20260620 21:49:28.648115 21366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:28.665176 21366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:28.666052 21366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:28.666270 21366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:28.673923 21366 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:43671
I20260620 21:49:28.673974 21432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:43671 every 8 connection(s)
I20260620 21:49:28.674957 21366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:28.678449 21433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:28.684245 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 21366
I20260620 21:49:28.684424 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:28.684643 21433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff: Bootstrap starting.
I20260620 21:49:28.684770 1646 external_mini_cluster.cc:1428] Setting key ffc38fd8677ea8e6bedfae724df81e6f
I20260620 21:49:28.687415 21433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:28.688201 21433 log.cc:824] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:28.690467 21433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff: No bootstrap required, opened a new log
I20260620 21:49:28.694794 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.686274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.1.155.190:43671)
0620 21:49:28.686973 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:28.686987 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:28.687023 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:49:28.687757 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:28.687776 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:28.688181 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:28.688518 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:28.689024 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.690087 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.691290 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.691818 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.691964 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.692806 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:28.692811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:28.692824 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:28.693282 (+ 458us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:28.693339 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:28.693359 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:28.693442 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:28.693633 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 21:49:28.693990 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":139,"threads_started":1}
I20260620 21:49:28.694667 21433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af78a8de91e48bca5e84c4dfadb1aff" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43671 } }
I20260620 21:49:28.694928 21433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:28.695003 21433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af78a8de91e48bca5e84c4dfadb1aff, State: Initialized, Role: FOLLOWER
I20260620 21:49:28.695447 21433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af78a8de91e48bca5e84c4dfadb1aff" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43671 } }
I20260620 21:49:28.695596 21433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:28.695647 21433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:28.695730 21433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:28.696616 21433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af78a8de91e48bca5e84c4dfadb1aff" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43671 } }
I20260620 21:49:28.696985 21433 leader_election.cc:302] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7af78a8de91e48bca5e84c4dfadb1aff; no voters:
I20260620 21:49:28.697353 21433 leader_election.cc:288] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:28.697409 21438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:28.697595 21438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [term 1 LEADER]: Becoming Leader. State: Replica: 7af78a8de91e48bca5e84c4dfadb1aff, State: Running, Role: LEADER
I20260620 21:49:28.697959 21438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af78a8de91e48bca5e84c4dfadb1aff" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43671 } }
I20260620 21:49:28.698606 21433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:28.699649 21440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7af78a8de91e48bca5e84c4dfadb1aff. Latest consensus state: current_term: 1 leader_uuid: "7af78a8de91e48bca5e84c4dfadb1aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af78a8de91e48bca5e84c4dfadb1aff" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43671 } } }
I20260620 21:49:28.699667 21439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7af78a8de91e48bca5e84c4dfadb1aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af78a8de91e48bca5e84c4dfadb1aff" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43671 } } }
I20260620 21:49:28.699797 21439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:28.699797 21440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:28.700263 21447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:28.703387 21447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:28.709192 21447 catalog_manager.cc:1347] Generated new cluster ID: 5996b44adb4b4aa2bb80c8c83231b7e7
I20260620 21:49:28.709280 21447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:28.750553 21447 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:28.751860 21447 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:28.762847 21447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff: Generated new TSK 0
I20260620 21:49:28.763725 21447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:28.773449 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.129:0
--local_ip_for_outbound_sockets=127.1.155.129
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:43671
--builtin_ntp_servers=127.1.155.148:37823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:49:28.890027 21457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:28.890293 21457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:28.890336 21457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:28.894501 21457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:28.894553 21457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:28.894644 21457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.129
I20260620 21:49:28.899567 21457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:37823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.155.129
--webserver_port=0
--tserver_master_addrs=127.1.155.190:43671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.129
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:28.900689 21457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:28.902086 21457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:28.911767 21463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:28.911743 21462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:28.911892 21457 server_base.cc:1034] running on GCE node
W20260620 21:49:28.911814 21465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:28.912357 21457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:28.913084 21457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:28.914289 21457 hybrid_clock.cc:648] HybridClock initialized: now 1781992168914253 us; error 47 us; skew 500 ppm
I20260620 21:49:28.916342 21457 webserver.cc:460] Webserver started at http://127.1.155.129:33709/ using document root <none> and password file <none>
I20260620 21:49:28.916954 21457 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:28.917062 21457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:28.917306 21457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:28.919530 21457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/data/instance:
uuid: "723fddf893934183a6860c7f0343d478"
format_stamp: "Formatted at 2026-06-20 21:49:28 on dist-test-slave-5bvn"
server_key: "ec45a59b2f35a59a4b4838042f07013d"
server_key_iv: "4372da78557644bac2e634dc69bbc455"
server_key_version: "encryptionkey@0"
I20260620 21:49:28.920121 21457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance:
uuid: "723fddf893934183a6860c7f0343d478"
format_stamp: "Formatted at 2026-06-20 21:49:28 on dist-test-slave-5bvn"
server_key: "ec45a59b2f35a59a4b4838042f07013d"
server_key_iv: "4372da78557644bac2e634dc69bbc455"
server_key_version: "encryptionkey@0"
I20260620 21:49:28.923935 21457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:49:28.926497 21471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:28.927718 21457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:49:28.927868 21457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/wal
uuid: "723fddf893934183a6860c7f0343d478"
format_stamp: "Formatted at 2026-06-20 21:49:28 on dist-test-slave-5bvn"
server_key: "ec45a59b2f35a59a4b4838042f07013d"
server_key_iv: "4372da78557644bac2e634dc69bbc455"
server_key_version: "encryptionkey@0"
I20260620 21:49:28.927985 21457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:28.943639 21457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:28.944346 21457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:28.944557 21457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:28.945164 21457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:28.946192 21457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:28.946267 21457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:28.946338 21457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:28.946373 21457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:28.955457 21457 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.129:46691
I20260620 21:49:28.955479 21584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.129:46691 every 8 connection(s)
I20260620 21:49:28.956421 21457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/data/info.pb
I20260620 21:49:28.960259 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 21457
I20260620 21:49:28.960414 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-0/wal/instance
I20260620 21:49:28.960798 1646 external_mini_cluster.cc:1428] Setting key c66f8fb1051f8fb06162122e052d2b17
I20260620 21:49:28.962860 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.130:0
--local_ip_for_outbound_sockets=127.1.155.130
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:43671
--builtin_ntp_servers=127.1.155.148:37823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:49:28.967134 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.958384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.129:44259 (local address 127.1.155.190:43671)
0620 21:49:28.958533 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:28.958536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:28.959521 (+ 985us) server_negotiation.cc:407] Connection header received
0620 21:49:28.961210 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:28.961213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:28.961271 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:28.961372 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:28.962569 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.963189 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.964832 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.965227 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.965350 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.966482 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:28.966489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:28.966495 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:28.966513 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:28.966539 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:28.966553 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:28.966637 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:28.966845 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 21:49:28.966919 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:49:28.967901 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:28.958747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:43671 (local address 127.1.155.129:44259)
0620 21:49:28.959372 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:28.959410 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:28.960931 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:28.961622 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:28.961629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:28.961712 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:28.962412 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.962432 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.963320 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.963326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.964708 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:28.964718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:28.965372 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:28.965375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:28.965526 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:28.966234 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:49:28.966235 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:28.966273 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:49:28.966334 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:28.966678 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:28.966689 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:28.966695 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:49:28.966813 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 21:49:28.967020 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":103,"threads_started":1}
I20260620 21:49:28.969112 21585 heartbeater.cc:347] Connected to a master server at 127.1.155.190:43671
I20260620 21:49:28.969362 21585 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:28.970017 21585 heartbeater.cc:511] Master 127.1.155.190:43671 requested a full tablet report, sending...
I20260620 21:49:28.971766 21397 ts_manager.cc:194] Registered new tserver with Master: 723fddf893934183a6860c7f0343d478 (127.1.155.129:46691)
I20260620 21:49:28.973484 21397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.155.129:44259
W20260620 21:49:29.074079 21588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:29.074360 21588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:29.074400 21588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:29.078483 21588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:29.078549 21588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:29.078606 21588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.130
I20260620 21:49:29.083331 21588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:37823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.155.130
--webserver_port=0
--tserver_master_addrs=127.1.155.190:43671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.130
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:29.084388 21588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:29.085747 21588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:29.095773 21596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:29.095865 21588 server_base.cc:1034] running on GCE node
W20260620 21:49:29.095769 21594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:29.095789 21593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:29.096421 21588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:29.097110 21588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:29.098359 21588 hybrid_clock.cc:648] HybridClock initialized: now 1781992169098328 us; error 35 us; skew 500 ppm
I20260620 21:49:29.100522 21588 webserver.cc:460] Webserver started at http://127.1.155.130:35907/ using document root <none> and password file <none>
I20260620 21:49:29.101115 21588 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:29.101212 21588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:29.101442 21588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:29.103530 21588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/data/instance:
uuid: "700d6f002ff94087b07bea17679f70a7"
format_stamp: "Formatted at 2026-06-20 21:49:29 on dist-test-slave-5bvn"
server_key: "32a86c9e12f8c9b28372c9124548240e"
server_key_iv: "03a00e92339412afdc25edbd69384902"
server_key_version: "encryptionkey@0"
I20260620 21:49:29.104085 21588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance:
uuid: "700d6f002ff94087b07bea17679f70a7"
format_stamp: "Formatted at 2026-06-20 21:49:29 on dist-test-slave-5bvn"
server_key: "32a86c9e12f8c9b28372c9124548240e"
server_key_iv: "03a00e92339412afdc25edbd69384902"
server_key_version: "encryptionkey@0"
I20260620 21:49:29.107923 21588 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 21:49:29.110664 21602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:29.111977 21588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:49:29.112141 21588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/wal
uuid: "700d6f002ff94087b07bea17679f70a7"
format_stamp: "Formatted at 2026-06-20 21:49:29 on dist-test-slave-5bvn"
server_key: "32a86c9e12f8c9b28372c9124548240e"
server_key_iv: "03a00e92339412afdc25edbd69384902"
server_key_version: "encryptionkey@0"
I20260620 21:49:29.112320 21588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:29.136829 21588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:29.137635 21588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:29.137887 21588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:29.138535 21588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:29.139581 21588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:29.139668 21588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:29.139773 21588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:29.139827 21588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:29.152735 21588 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.130:40149
I20260620 21:49:29.152990 21715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.130:40149 every 8 connection(s)
I20260620 21:49:29.153940 21588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/data/info.pb
I20260620 21:49:29.160974 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 21588
I20260620 21:49:29.161165 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/wal/instance
I20260620 21:49:29.161662 1646 external_mini_cluster.cc:1428] Setting key 188246b438d2e398a958e3386f620e24
I20260620 21:49:29.164945 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.155.131:0
--local_ip_for_outbound_sockets=127.1.155.131
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:43671
--builtin_ntp_servers=127.1.155.148:37823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:49:29.167389 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.156177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.130:36907 (local address 127.1.155.190:43671)
0620 21:49:29.156365 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:29.156370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:29.157785 (+ 1415us) server_negotiation.cc:407] Connection header received
0620 21:49:29.159629 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:29.159632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:29.159682 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:29.159763 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:29.162274 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.162864 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.164754 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.165124 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.165258 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.166829 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:29.166832 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:29.166834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:29.166848 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:29.166869 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:29.166877 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:29.166948 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:29.167207 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 21:49:29.167270 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 21:49:29.168650 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.156679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:43671 (local address 127.1.155.130:36907)
0620 21:49:29.157606 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:29.157653 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:29.159366 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:29.159963 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:29.159970 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:29.160703 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:29.161983 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:29.162014 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.163029 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.163032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:29.164477 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:29.164495 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.165360 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.165364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:29.165587 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.166415 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:49:29.166418 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:29.166490 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:49:29.166595 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:29.166979 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:29.166991 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:29.166998 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 21:49:29.167137 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 21:49:29.167500 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":127,"threads_started":1}
I20260620 21:49:29.170306 21716 heartbeater.cc:347] Connected to a master server at 127.1.155.190:43671
I20260620 21:49:29.170562 21716 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:29.171144 21716 heartbeater.cc:511] Master 127.1.155.190:43671 requested a full tablet report, sending...
I20260620 21:49:29.172593 21397 ts_manager.cc:194] Registered new tserver with Master: 700d6f002ff94087b07bea17679f70a7 (127.1.155.130:40149)
I20260620 21:49:29.173431 21397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.155.130:36907
W20260620 21:49:29.280033 21719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:29.280323 21719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:29.280365 21719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:29.284585 21719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:29.284680 21719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:29.284789 21719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.155.131
I20260620 21:49:29.289669 21719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:37823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.155.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.155.131
--webserver_port=0
--tserver_master_addrs=127.1.155.190:43671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.155.131
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:29.290796 21719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:29.292249 21719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:29.302017 21727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:29.302026 21725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:29.302276 21719 server_base.cc:1034] running on GCE node
W20260620 21:49:29.302049 21724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:29.302829 21719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:29.303539 21719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:29.304719 21719 hybrid_clock.cc:648] HybridClock initialized: now 1781992169304683 us; error 49 us; skew 500 ppm
I20260620 21:49:29.310047 21719 webserver.cc:460] Webserver started at http://127.1.155.131:35055/ using document root <none> and password file <none>
I20260620 21:49:29.310650 21719 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:29.310719 21719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:29.310935 21719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:29.313105 21719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/data/instance:
uuid: "02a34059b5dc494fa2fc396606065d7a"
format_stamp: "Formatted at 2026-06-20 21:49:29 on dist-test-slave-5bvn"
server_key: "1c4a36deb69f149439da44ae890ec398"
server_key_iv: "9ddbb3ee290173ef699d238ff105de54"
server_key_version: "encryptionkey@0"
I20260620 21:49:29.313676 21719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance:
uuid: "02a34059b5dc494fa2fc396606065d7a"
format_stamp: "Formatted at 2026-06-20 21:49:29 on dist-test-slave-5bvn"
server_key: "1c4a36deb69f149439da44ae890ec398"
server_key_iv: "9ddbb3ee290173ef699d238ff105de54"
server_key_version: "encryptionkey@0"
I20260620 21:49:29.317618 21719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 21:49:29.320099 21733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:29.321319 21719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:49:29.321460 21719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/wal
uuid: "02a34059b5dc494fa2fc396606065d7a"
format_stamp: "Formatted at 2026-06-20 21:49:29 on dist-test-slave-5bvn"
server_key: "1c4a36deb69f149439da44ae890ec398"
server_key_iv: "9ddbb3ee290173ef699d238ff105de54"
server_key_version: "encryptionkey@0"
I20260620 21:49:29.321579 21719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:29.359836 21719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:29.360572 21719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:29.360785 21719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:29.361393 21719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:49:29.362360 21719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:49:29.362447 21719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:29.362519 21719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:49:29.362574 21719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:29.371536 21719 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.131:34149
I20260620 21:49:29.371557 21846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.131:34149 every 8 connection(s)
I20260620 21:49:29.372447 21719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/data/info.pb
I20260620 21:49:29.372947 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 21719
I20260620 21:49:29.373082 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/wal/instance
I20260620 21:49:29.373412 1646 external_mini_cluster.cc:1428] Setting key 36601cf49cb53ebe13f06e84a324e9b2
I20260620 21:49:29.382190 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.374418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:55539 (local address 127.1.155.190:43671)
0620 21:49:29.374550 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:29.374554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:29.375820 (+ 1266us) server_negotiation.cc:407] Connection header received
0620 21:49:29.377224 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:29.377227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:29.377284 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:29.377369 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:29.378412 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.378953 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.380113 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.380347 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.380463 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.381617 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:29.381621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:29.381624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:29.381637 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:29.381664 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:29.381672 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:29.381752 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:29.381970 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 21:49:29.382037 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:49:29.383009 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.374935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:43671 (local address 127.1.155.131:55539)
0620 21:49:29.375677 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:29.375729 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:29.377012 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:29.377568 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:29.377575 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:29.377660 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:29.378263 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:29.378277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.379103 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.379107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:29.379980 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:29.379989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.380503 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.380507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:29.380678 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.381337 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:49:29.381338 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:29.381377 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:49:29.381442 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:29.381754 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:29.381766 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:29.381774 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 21:49:29.381970 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 21:49:29.382248 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":135,"threads_started":1}
I20260620 21:49:29.384147 21847 heartbeater.cc:347] Connected to a master server at 127.1.155.190:43671
I20260620 21:49:29.384357 21847 heartbeater.cc:464] Registering TS with master...
I20260620 21:49:29.384860 21847 heartbeater.cc:511] Master 127.1.155.190:43671 requested a full tablet report, sending...
I20260620 21:49:29.385886 21397 ts_manager.cc:194] Registered new tserver with Master: 02a34059b5dc494fa2fc396606065d7a (127.1.155.131:34149)
I20260620 21:49:29.386698 21397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.155.131:55539
I20260620 21:49:29.387670 1646 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:49:29.393272 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.389714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.1.155.190:43671)
0620 21:49:29.389893 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:29.389897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:29.390036 (+ 139us) server_negotiation.cc:407] Connection header received
0620 21:49:29.390141 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:29.390143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:29.390179 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:29.390245 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:29.390506 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.390984 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.391803 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.392003 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.392078 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.392938 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:29.392941 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:29.392943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:29.392953 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:29.392968 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:29.392975 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:29.393033 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:29.393129 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 21:49:29.393172 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:49:29.396162 21397 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:49:29.399471 21397 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:49:29.413496 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.408844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39448 (local address 127.1.155.129:46691)
0620 21:49:29.409422 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:29.409432 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:29.409453 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:49:29.409522 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:29.409526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:29.409694 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:29.409852 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:29.410041 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.410799 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.412271 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.412811 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.412944 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.413143 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:29.413230 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 21:49:29.413325 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":123,"threads_started":1}
I20260620 21:49:29.413491 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.408440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:46691 (local address 127.0.0.1:39448)
0620 21:49:29.409247 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:29.409260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:29.409336 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:29.409812 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:29.409815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:29.409855 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:29.409941 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:29.409951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.410928 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.410937 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:29.412145 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:29.412154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.412953 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.412957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:29.413059 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.413097 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 21:49:29.413204 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 21:49:29.413325 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":7,"thread_start_us":82,"threads_started":1}
I20260620 21:49:29.413971 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.407893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:40149 (local address 127.0.0.1:37774)
0620 21:49:29.408577 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:29.408591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:29.408664 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:29.409652 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:29.409655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:29.409674 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:29.409769 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:29.409775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.411130 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.411134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:29.412293 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:29.412305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.413483 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.413487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:29.413634 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.413659 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:49:29.413724 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 21:49:29.413787 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":49,"thread_start_us":95,"threads_started":1}
I20260620 21:49:29.414170 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.407702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.131:34149 (local address 127.0.0.1:43728)
0620 21:49:29.408147 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:29.408197 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:29.408332 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:29.409170 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:29.409176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:29.409200 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:29.409349 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:29.409357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.410799 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.410804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:29.412216 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:29.412227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.413483 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.413487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:29.413873 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.413896 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:49:29.413968 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 21:49:29.414046 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":30,"thread_start_us":74,"threads_started":1}
I20260620 21:49:29.414263 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.407848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.1.155.131:34149)
0620 21:49:29.408521 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:29.408533 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:29.408556 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:49:29.408634 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:29.408640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:29.408842 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:29.409014 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:29.409457 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.410664 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.412365 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.413153 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.413257 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.413418 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:29.414018 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 21:49:29.414115 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":215,"threads_started":1}
I20260620 21:49:29.416600 21519 tablet_service.cc:1467] Processing CreateTablet for tablet 08ae3b2c51b54e8091e0f9b8d9197057 (DEFAULT_TABLE table=test-table [id=3d549b35542d46c78a50dd3ddda952a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:29.416704 21781 tablet_service.cc:1467] Processing CreateTablet for tablet 08ae3b2c51b54e8091e0f9b8d9197057 (DEFAULT_TABLE table=test-table [id=3d549b35542d46c78a50dd3ddda952a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:29.417603 21781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08ae3b2c51b54e8091e0f9b8d9197057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:29.417603 21519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08ae3b2c51b54e8091e0f9b8d9197057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:29.418286 21862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.408499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.1.155.130:40149)
0620 21:49:29.409042 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:29.409054 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:29.409078 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:49:29.409149 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:29.409154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:29.409491 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:29.409648 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:29.409918 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.411016 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.412415 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.413153 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.413257 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.413418 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:29.418010 (+ 4592us) server_negotiation.cc:299] Negotiation successful
0620 21:49:29.418126 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":97,"threads_started":1}
I20260620 21:49:29.421012 21650 tablet_service.cc:1467] Processing CreateTablet for tablet 08ae3b2c51b54e8091e0f9b8d9197057 (DEFAULT_TABLE table=test-table [id=3d549b35542d46c78a50dd3ddda952a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:49:29.422118 21650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08ae3b2c51b54e8091e0f9b8d9197057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:29.424546 21864 tablet_bootstrap.cc:492] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a: Bootstrap starting.
I20260620 21:49:29.425108 21865 tablet_bootstrap.cc:492] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478: Bootstrap starting.
I20260620 21:49:29.426925 21864 tablet_bootstrap.cc:654] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:29.427295 21865 tablet_bootstrap.cc:654] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:29.427855 21864 log.cc:824] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:29.428043 21865 log.cc:824] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:29.429949 21864 tablet_bootstrap.cc:492] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a: No bootstrap required, opened a new log
I20260620 21:49:29.430073 21868 tablet_bootstrap.cc:492] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7: Bootstrap starting.
I20260620 21:49:29.430107 21865 tablet_bootstrap.cc:492] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478: No bootstrap required, opened a new log
I20260620 21:49:29.430143 21864 ts_tablet_manager.cc:1397] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:49:29.430244 21865 ts_tablet_manager.cc:1397] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:49:29.432202 21868 tablet_bootstrap.cc:654] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:29.433009 21868 log.cc:824] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:29.433645 21865 raft_consensus.cc:348] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a34059b5dc494fa2fc396606065d7a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34149 } } peers { permanent_uuid: "700d6f002ff94087b07bea17679f70a7" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 40149 } } peers { permanent_uuid: "723fddf893934183a6860c7f0343d478" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 46691 } }
I20260620 21:49:29.433990 21865 raft_consensus.cc:374] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:29.434082 21865 raft_consensus.cc:729] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723fddf893934183a6860c7f0343d478, State: Initialized, Role: FOLLOWER
I20260620 21:49:29.434343 21864 raft_consensus.cc:348] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a34059b5dc494fa2fc396606065d7a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34149 } } peers { permanent_uuid: "700d6f002ff94087b07bea17679f70a7" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 40149 } } peers { permanent_uuid: "723fddf893934183a6860c7f0343d478" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 46691 } }
I20260620 21:49:29.434594 21864 raft_consensus.cc:374] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:29.434563 21865 consensus_queue.cc:260] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a34059b5dc494fa2fc396606065d7a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34149 } } peers { permanent_uuid: "700d6f002ff94087b07bea17679f70a7" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 40149 } } peers { permanent_uuid: "723fddf893934183a6860c7f0343d478" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 46691 } }
I20260620 21:49:29.434674 21864 raft_consensus.cc:729] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a34059b5dc494fa2fc396606065d7a, State: Initialized, Role: FOLLOWER
I20260620 21:49:29.435142 21868 tablet_bootstrap.cc:492] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7: No bootstrap required, opened a new log
I20260620 21:49:29.435122 21864 consensus_queue.cc:260] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a34059b5dc494fa2fc396606065d7a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34149 } } peers { permanent_uuid: "700d6f002ff94087b07bea17679f70a7" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 40149 } } peers { permanent_uuid: "723fddf893934183a6860c7f0343d478" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 46691 } }
I20260620 21:49:29.435230 21585 heartbeater.cc:503] Master 127.1.155.190:43671 was elected leader, sending a full tablet report...
I20260620 21:49:29.435408 21868 ts_tablet_manager.cc:1397] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:49:29.435609 21865 ts_tablet_manager.cc:1428] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 21:49:29.436125 21864 ts_tablet_manager.cc:1428] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:49:29.436494 21847 heartbeater.cc:503] Master 127.1.155.190:43671 was elected leader, sending a full tablet report...
I20260620 21:49:29.439657 21868 raft_consensus.cc:348] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a34059b5dc494fa2fc396606065d7a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34149 } } peers { permanent_uuid: "700d6f002ff94087b07bea17679f70a7" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 40149 } } peers { permanent_uuid: "723fddf893934183a6860c7f0343d478" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 46691 } }
I20260620 21:49:29.440062 21868 raft_consensus.cc:374] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:29.440168 21868 raft_consensus.cc:729] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700d6f002ff94087b07bea17679f70a7, State: Initialized, Role: FOLLOWER
I20260620 21:49:29.440605 21868 consensus_queue.cc:260] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a34059b5dc494fa2fc396606065d7a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34149 } } peers { permanent_uuid: "700d6f002ff94087b07bea17679f70a7" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 40149 } } peers { permanent_uuid: "723fddf893934183a6860c7f0343d478" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 46691 } }
I20260620 21:49:29.441272 21716 heartbeater.cc:503] Master 127.1.155.190:43671 was elected leader, sending a full tablet report...
I20260620 21:49:29.441505 21868 ts_tablet_manager.cc:1428] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 21:49:29.457621 21586 tablet.cc:2369] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:49:29.537748 21871 raft_consensus.cc:482] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:49:29.537999 21871 raft_consensus.cc:504] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a34059b5dc494fa2fc396606065d7a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34149 } } peers { permanent_uuid: "700d6f002ff94087b07bea17679f70a7" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 40149 } } peers { permanent_uuid: "723fddf893934183a6860c7f0343d478" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 46691 } }
I20260620 21:49:29.539238 21871 leader_election.cc:288] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 700d6f002ff94087b07bea17679f70a7 (127.1.155.130:40149), 723fddf893934183a6860c7f0343d478 (127.1.155.129:46691)
I20260620 21:49:29.542848 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.539431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.130:40149 (local address 127.1.155.131:47029)
0620 21:49:29.539581 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:29.539599 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:29.539696 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:29.539995 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:29.539998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:29.540015 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:29.540175 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:29.540181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.540918 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.540925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:29.541743 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:29.541752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.542406 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.542409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:29.542520 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.542548 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:49:29.542649 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 21:49:29.542720 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"spinlock_wait_cycles":19200}
I20260620 21:49:29.542848 21862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.539576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:47029 (local address 127.1.155.130:40149)
0620 21:49:29.539717 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:29.539724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:29.539743 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:49:29.539805 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:29.539808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:29.539861 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:29.539969 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:29.540295 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.540802 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.541890 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.542278 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.542415 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.542465 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:29.542664 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 21:49:29.542727 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:49:29.543216 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.539559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:46691 (local address 127.1.155.131:38085)
0620 21:49:29.539930 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:29.539941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:29.540074 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:29.540342 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:29.540345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:29.540360 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:49:29.540484 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:29.540490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.541205 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.541210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:29.542058 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:29.542067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.542729 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.542733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:29.542834 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.542857 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:49:29.542984 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 21:49:29.543075 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"spinlock_wait_cycles":768,"thread_start_us":136,"threads_started":1}
I20260620 21:49:29.543305 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.539702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.155.131:38085 (local address 127.1.155.129:46691)
0620 21:49:29.539834 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:29.539838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:29.540024 (+ 186us) server_negotiation.cc:407] Connection header received
0620 21:49:29.540187 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:29.540192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:29.540239 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:29.540330 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:49:29.540602 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.541087 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.542183 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.542603 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.542739 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.542784 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:29.543113 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 21:49:29.543181 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:49:29.543617 21670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08ae3b2c51b54e8091e0f9b8d9197057" candidate_uuid: "02a34059b5dc494fa2fc396606065d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "700d6f002ff94087b07bea17679f70a7" is_pre_election: true
I20260620 21:49:29.543834 21539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08ae3b2c51b54e8091e0f9b8d9197057" candidate_uuid: "02a34059b5dc494fa2fc396606065d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723fddf893934183a6860c7f0343d478" is_pre_election: true
I20260620 21:49:29.543939 21670 raft_consensus.cc:2454] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02a34059b5dc494fa2fc396606065d7a in term 0.
I20260620 21:49:29.544126 21539 raft_consensus.cc:2454] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02a34059b5dc494fa2fc396606065d7a in term 0.
I20260620 21:49:29.544462 21736 leader_election.cc:302] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a34059b5dc494fa2fc396606065d7a, 700d6f002ff94087b07bea17679f70a7; no voters:
I20260620 21:49:29.544770 21871 raft_consensus.cc:2781] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:49:29.544847 21871 raft_consensus.cc:482] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:49:29.544878 21871 raft_consensus.cc:3037] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:29.545910 21871 raft_consensus.cc:504] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a34059b5dc494fa2fc396606065d7a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34149 } } peers { permanent_uuid: "700d6f002ff94087b07bea17679f70a7" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 40149 } } peers { permanent_uuid: "723fddf893934183a6860c7f0343d478" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 46691 } }
I20260620 21:49:29.546260 21871 leader_election.cc:288] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [CANDIDATE]: Term 1 election: Requested vote from peers 700d6f002ff94087b07bea17679f70a7 (127.1.155.130:40149), 723fddf893934183a6860c7f0343d478 (127.1.155.129:46691)
I20260620 21:49:29.546639 21539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08ae3b2c51b54e8091e0f9b8d9197057" candidate_uuid: "02a34059b5dc494fa2fc396606065d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723fddf893934183a6860c7f0343d478"
I20260620 21:49:29.546648 21670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08ae3b2c51b54e8091e0f9b8d9197057" candidate_uuid: "02a34059b5dc494fa2fc396606065d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "700d6f002ff94087b07bea17679f70a7"
I20260620 21:49:29.546762 21670 raft_consensus.cc:3037] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:29.546762 21539 raft_consensus.cc:3037] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:29.547979 21539 raft_consensus.cc:2454] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02a34059b5dc494fa2fc396606065d7a in term 1.
I20260620 21:49:29.547979 21670 raft_consensus.cc:2454] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02a34059b5dc494fa2fc396606065d7a in term 1.
I20260620 21:49:29.548285 21737 leader_election.cc:302] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a34059b5dc494fa2fc396606065d7a, 723fddf893934183a6860c7f0343d478; no voters:
I20260620 21:49:29.548481 21871 raft_consensus.cc:2781] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:29.548730 21871 raft_consensus.cc:686] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [term 1 LEADER]: Becoming Leader. State: Replica: 02a34059b5dc494fa2fc396606065d7a, State: Running, Role: LEADER
I20260620 21:49:29.549116 21871 consensus_queue.cc:237] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a34059b5dc494fa2fc396606065d7a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34149 } } peers { permanent_uuid: "700d6f002ff94087b07bea17679f70a7" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 40149 } } peers { permanent_uuid: "723fddf893934183a6860c7f0343d478" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 46691 } }
I20260620 21:49:29.552466 21397 catalog_manager.cc:5495] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 02a34059b5dc494fa2fc396606065d7a (127.1.155.131). New cstate: current_term: 1 leader_uuid: "02a34059b5dc494fa2fc396606065d7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a34059b5dc494fa2fc396606065d7a" member_type: VOTER last_known_addr { host: "127.1.155.131" port: 34149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "700d6f002ff94087b07bea17679f70a7" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 40149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "723fddf893934183a6860c7f0343d478" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 46691 } health_report { overall_health: UNKNOWN } } }
I20260620 21:49:29.565986 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.562545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.1.155.131:34149)
0620 21:49:29.562659 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:29.562662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:29.562697 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:49:29.562835 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:29.562837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:29.562890 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:29.562984 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:49:29.563206 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.563708 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.564519 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:29.564750 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:29.564837 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:29.565083 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:49:29.565587 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:49:29.565694 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:29.565748 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 21:49:29.565790 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 21:49:29.571758 21539 raft_consensus.cc:1261] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478 [term 1 FOLLOWER]: Refusing update from remote peer 02a34059b5dc494fa2fc396606065d7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:49:29.571758 21670 raft_consensus.cc:1261] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7 [term 1 FOLLOWER]: Refusing update from remote peer 02a34059b5dc494fa2fc396606065d7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:49:29.572484 21871 consensus_queue.cc:1035] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "700d6f002ff94087b07bea17679f70a7" member_type: VOTER last_known_addr { host: "127.1.155.130" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:49:29.572695 21874 consensus_queue.cc:1035] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "723fddf893934183a6860c7f0343d478" member_type: VOTER last_known_addr { host: "127.1.155.129" port: 46691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:49:29.583985 21878 mvcc.cc:205] Tried to move back new op lower bound from 7299039926558658560 to 7299039926476414976. Current Snapshot: MvccSnapshot[applied={T|T < 7299039926558658560 or (T in {7299039926558658560})}]
I20260620 21:49:29.588965 21396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:49:29.589156 21396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:49:29.591840 21396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7af78a8de91e48bca5e84c4dfadb1aff: Sending DeleteTablet for 3 replicas of tablet 08ae3b2c51b54e8091e0f9b8d9197057
I20260620 21:49:29.592670 21781 tablet_service.cc:1514] Processing DeleteTablet for tablet 08ae3b2c51b54e8091e0f9b8d9197057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:29 UTC) from {username='slave'} at 127.0.0.1:43728
I20260620 21:49:29.592782 21650 tablet_service.cc:1514] Processing DeleteTablet for tablet 08ae3b2c51b54e8091e0f9b8d9197057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:29 UTC) from {username='slave'} at 127.0.0.1:37774
I20260620 21:49:29.592931 21519 tablet_service.cc:1514] Processing DeleteTablet for tablet 08ae3b2c51b54e8091e0f9b8d9197057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:49:29 UTC) from {username='slave'} at 127.0.0.1:39448
I20260620 21:49:29.593248 21888 tablet_replica.cc:330] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7: stopping tablet replica
I20260620 21:49:29.593379 21887 tablet_replica.cc:330] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a: stopping tablet replica
I20260620 21:49:29.593653 21888 raft_consensus.cc:2229] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:49:29.593716 21887 raft_consensus.cc:2229] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:49:29.594117 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 21457
I20260620 21:49:29.594110 21889 tablet_replica.cc:330] T 08ae3b2c51b54e8091e0f9b8d9197057 P 723fddf893934183a6860c7f0343d478: stopping tablet replica
I20260620 21:49:29.594110 21888 raft_consensus.cc:2258] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:49:29.596757 21888 ts_tablet_manager.cc:1905] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:49:29.599711 21888 ts_tablet_manager.cc:1918] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:49:29.599819 21888 log.cc:1196] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-1/wal/wals/08ae3b2c51b54e8091e0f9b8d9197057
I20260620 21:49:29.600133 21888 ts_tablet_manager.cc:1939] T 08ae3b2c51b54e8091e0f9b8d9197057 P 700d6f002ff94087b07bea17679f70a7: Deleting consensus metadata
W20260620 21:49:29.601338 21384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.155.129:46691 (error 108)
I20260620 21:49:29.601447 21383 catalog_manager.cc:4841] TS 700d6f002ff94087b07bea17679f70a7 (127.1.155.130:40149): tablet 08ae3b2c51b54e8091e0f9b8d9197057 (table test-table [id=3d549b35542d46c78a50dd3ddda952a5]) successfully deleted
I20260620 21:49:29.601816 21887 raft_consensus.cc:2258] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:49:29.602015 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 21588
I20260620 21:49:29.603700 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:29.603309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.129:46691 (local address 127.0.0.1:39456)
0620 21:49:29.603518 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:29.603596 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.129:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260620 21:49:29.603950 21384 catalog_manager.cc:4584] TS 723fddf893934183a6860c7f0343d478 (127.1.155.129:46691): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08ae3b2c51b54e8091e0f9b8d9197057: Network error: Client connection negotiation failed: client connection to 127.1.155.129:46691: connect: Connection refused (error 111)
I20260620 21:49:29.604126 21887 ts_tablet_manager.cc:1905] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:49:29.606829 21887 ts_tablet_manager.cc:1918] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:49:29.606935 21887 log.cc:1196] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a: Deleting WAL directory at /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781991920067886-1646-0/minicluster-data/ts-2/wal/wals/08ae3b2c51b54e8091e0f9b8d9197057
I20260620 21:49:29.607240 21887 ts_tablet_manager.cc:1939] T 08ae3b2c51b54e8091e0f9b8d9197057 P 02a34059b5dc494fa2fc396606065d7a: Deleting consensus metadata
I20260620 21:49:29.608521 21382 catalog_manager.cc:4841] TS 02a34059b5dc494fa2fc396606065d7a (127.1.155.131:34149): tablet 08ae3b2c51b54e8091e0f9b8d9197057 (table test-table [id=3d549b35542d46c78a50dd3ddda952a5]) successfully deleted
I20260620 21:49:29.609411 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 21719
I20260620 21:49:29.616534 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 21366
2026-06-20T21:49:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1148 ms)
[----------] 6 tests from EncryptionPolicyTest (10488 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:29 dist-test-slave-5bvn krb5kdc[21893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:29 dist-test-slave-5bvn krb5kdc[21893](info): set up 2 sockets
Jun 20 21:49:29 dist-test-slave-5bvn krb5kdc[21893](info): commencing operation
krb5kdc: starting...
W20260620 21:49:30.678053 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:30 dist-test-slave-5bvn krb5kdc[21893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:49:30.804587 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:30.915870 21906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:30.916172 21906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:30.916215 21906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:30.920447 21906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:30.920523 21906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:30.920543 21906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:30.920562 21906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:30.920578 21906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:30.926098 21906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45693
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:30.927150 21906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:30.928104 21906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:30.937153 21911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:30.937335 21912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:30.937140 21914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:30.937239 21906 server_base.cc:1034] running on GCE node
I20260620 21:49:30.938082 21906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:30.939244 21906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:30.940459 21906 hybrid_clock.cc:648] HybridClock initialized: now 1781992170940445 us; error 26 us; skew 500 ppm
Jun 20 21:49:30 dist-test-slave-5bvn krb5kdc[21893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:30.943398 21906 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:49:30.944285 21906 webserver.cc:460] Webserver started at http://127.0.0.1:45783/ using document root <none> and password file <none>
I20260620 21:49:30.944859 21906 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:30.944957 21906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:30.945201 21906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:30.947311 21906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "2af9f7d05c634d9f84e1360dc9eba812"
format_stamp: "Formatted at 2026-06-20 21:49:30 on dist-test-slave-5bvn"
server_key: "08a42118afe9621a1bfde6fca7f4666d"
server_key_iv: "e9b19b7449a5bf6268a22f30b29b40df"
server_key_version: "encryptionkey@0"
I20260620 21:49:30.947880 21906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "2af9f7d05c634d9f84e1360dc9eba812"
format_stamp: "Formatted at 2026-06-20 21:49:30 on dist-test-slave-5bvn"
server_key: "08a42118afe9621a1bfde6fca7f4666d"
server_key_iv: "e9b19b7449a5bf6268a22f30b29b40df"
server_key_version: "encryptionkey@0"
I20260620 21:49:30.951715 21906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 21:49:30.954452 21921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:30.955760 21906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:49:30.955922 21906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "2af9f7d05c634d9f84e1360dc9eba812"
format_stamp: "Formatted at 2026-06-20 21:49:30 on dist-test-slave-5bvn"
server_key: "08a42118afe9621a1bfde6fca7f4666d"
server_key_iv: "e9b19b7449a5bf6268a22f30b29b40df"
server_key_version: "encryptionkey@0"
I20260620 21:49:30.956063 21906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:30.993480 21906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:30.997066 21906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:30.997298 21906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:31.004321 21906 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45693
I20260620 21:49:31.004346 21973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45693 every 8 connection(s)
I20260620 21:49:31.005295 21906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:31.007943 21974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:31.011588 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 21906
I20260620 21:49:31.011750 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:31.012029 1646 external_mini_cluster.cc:1428] Setting key 228e0b3285c3483031d7ccd68dde4c47
I20260620 21:49:31.013943 21974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812: Bootstrap starting.
I20260620 21:49:31.017293 21974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:31.018239 21974 log.cc:824] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:31.020963 21974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812: No bootstrap required, opened a new log
Jun 20 21:49:31 dist-test-slave-5bvn krb5kdc[21893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 21:49:31.025803 21974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af9f7d05c634d9f84e1360dc9eba812" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } }
I20260620 21:49:31.026147 21974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:31.026232 21974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af9f7d05c634d9f84e1360dc9eba812, State: Initialized, Role: FOLLOWER
I20260620 21:49:31.026744 21974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af9f7d05c634d9f84e1360dc9eba812" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } }
I20260620 21:49:31.026870 21974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:31.026955 21974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:31.027056 21974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:31.028069 21974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af9f7d05c634d9f84e1360dc9eba812" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } }
I20260620 21:49:31.028453 21974 leader_election.cc:302] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2af9f7d05c634d9f84e1360dc9eba812; no voters:
I20260620 21:49:31.028843 21974 leader_election.cc:288] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:31.028921 21979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:31.029110 21979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [term 1 LEADER]: Becoming Leader. State: Replica: 2af9f7d05c634d9f84e1360dc9eba812, State: Running, Role: LEADER
I20260620 21:49:31.029206 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:31.013476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60281 (local address 127.0.0.1:45693)
0620 21:49:31.013902 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:31.013922 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:31.013965 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:49:31.014764 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:31.014786 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:31.015248 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:31.015601 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:31.016481 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:31.018226 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:31.019466 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:31.020190 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:31.020359 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:31.023468 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:31.023474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:31.023493 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:31.023525 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:31.025608 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:31.026258 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:31.026269 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:31.026274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:31.026431 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:31.026799 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:31.026802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:31.026804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:31.027367 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:31.027621 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:31.027919 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 21:49:31.028154 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":103,"threads_started":1}
I20260620 21:49:31.029460 21979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af9f7d05c634d9f84e1360dc9eba812" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } }
I20260620 21:49:31.030169 21974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:31.030654 21981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2af9f7d05c634d9f84e1360dc9eba812. Latest consensus state: current_term: 1 leader_uuid: "2af9f7d05c634d9f84e1360dc9eba812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af9f7d05c634d9f84e1360dc9eba812" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } } }
I20260620 21:49:31.030771 21981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:31.031543 21980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2af9f7d05c634d9f84e1360dc9eba812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af9f7d05c634d9f84e1360dc9eba812" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } } }
I20260620 21:49:31.031634 21980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:31.032354 21983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:31.032874 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:31.035878 21983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:31.042146 21983 catalog_manager.cc:1347] Generated new cluster ID: 4ec304e8558449f59e48ab404e2a8318
I20260620 21:49:31.042331 21983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:31.044121 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:31.035180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60325 (local address 127.0.0.1:45693)
0620 21:49:31.035300 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:31.035304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:31.035516 (+ 212us) server_negotiation.cc:407] Connection header received
0620 21:49:31.035635 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:31.035638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:31.035691 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:31.035796 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:31.036646 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:31.037444 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:31.038418 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:31.038669 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:31.038794 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:31.040445 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:31.040450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:31.040453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:31.040482 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:31.042274 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:31.042785 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:31.042789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:31.042791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:31.042879 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:31.043228 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:31.043232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:31.043234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:31.043415 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:31.043574 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:31.043859 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 21:49:31.043953 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 21:49:31.063719 21983 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:31.064719 21983 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:31.075510 21983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2af9f7d05c634d9f84e1360dc9eba812: Generated new TSK 0
I20260620 21:49:31.076388 21983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:31.087905 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 21906
W20260620 21:49:31.093586 21891 connection.cc:535] client connection to 127.0.0.1:45693 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45693): Connection reset by peer (error 104)
2026-06-20T21:49:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:31 dist-test-slave-5bvn krb5kdc[22005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:31 dist-test-slave-5bvn krb5kdc[22005](info): set up 2 sockets
Jun 20 21:49:31 dist-test-slave-5bvn krb5kdc[22005](info): commencing operation
krb5kdc: starting...
W20260620 21:49:32.143800 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:32 dist-test-slave-5bvn krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:49:32.277297 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:32.392313 22018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:32.392575 22018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:32.392617 22018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:32.396683 22018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:32.396744 22018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:32.396762 22018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:32.396780 22018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:32.396795 22018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:32.401904 22018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45183
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:32.402859 22018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:32.403872 22018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:32.412432 22023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:32.412424 22026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:32.412549 22024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:32.412652 22018 server_base.cc:1034] running on GCE node
I20260620 21:49:32.413528 22018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:32.414824 22018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:32.416014 22018 hybrid_clock.cc:648] HybridClock initialized: now 1781992172415995 us; error 34 us; skew 500 ppm
Jun 20 21:49:32 dist-test-slave-5bvn krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:32.419178 22018 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:49:32.420325 22018 webserver.cc:460] Webserver started at http://127.0.0.1:40833/ using document root <none> and password file <none>
I20260620 21:49:32.420958 22018 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:32.421066 22018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:32.421320 22018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:32.423594 22018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "dfe8dd75d8ff44f5bd62d0c35a67ba0a"
format_stamp: "Formatted at 2026-06-20 21:49:32 on dist-test-slave-5bvn"
server_key: "13a5f930b06889b347943e98b484bb9d"
server_key_iv: "b08ad4ff4fa5013687d340cd3685d572"
server_key_version: "encryptionkey@0"
I20260620 21:49:32.424207 22018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "dfe8dd75d8ff44f5bd62d0c35a67ba0a"
format_stamp: "Formatted at 2026-06-20 21:49:32 on dist-test-slave-5bvn"
server_key: "13a5f930b06889b347943e98b484bb9d"
server_key_iv: "b08ad4ff4fa5013687d340cd3685d572"
server_key_version: "encryptionkey@0"
I20260620 21:49:32.428287 22018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:49:32.430923 22033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:32.432255 22018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:49:32.432425 22018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "dfe8dd75d8ff44f5bd62d0c35a67ba0a"
format_stamp: "Formatted at 2026-06-20 21:49:32 on dist-test-slave-5bvn"
server_key: "13a5f930b06889b347943e98b484bb9d"
server_key_iv: "b08ad4ff4fa5013687d340cd3685d572"
server_key_version: "encryptionkey@0"
I20260620 21:49:32.432552 22018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:32.457367 22018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:32.461143 22018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:32.461375 22018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:32.469129 22018 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45183
I20260620 21:49:32.469152 22085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45183 every 8 connection(s)
I20260620 21:49:32.470186 22018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:32.472873 22086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:32.474452 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 22018
I20260620 21:49:32.474599 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:32.474920 1646 external_mini_cluster.cc:1428] Setting key 398fd31a9a42a3996dbe14b29eae91b7
I20260620 21:49:32.479034 22086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a: Bootstrap starting.
I20260620 21:49:32.481273 22086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:32.482090 22086 log.cc:824] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:32.484130 22086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a: No bootstrap required, opened a new log
I20260620 21:49:32.484953 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:32.476743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56937 (local address 127.0.0.1:45183)
0620 21:49:32.477268 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:32.477285 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:32.477328 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:49:32.478281 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:32.478301 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:32.478657 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:32.479065 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:32.479488 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:32.480450 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:32.481370 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:32.481929 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:32.482105 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:32.483043 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:32.483049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:32.483073 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:32.483472 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:32.483527 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:32.483549 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:32.483707 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:32.484028 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 21:49:32.484167 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":149,"threads_started":1}
I20260620 21:49:32.488029 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:32.488159 22086 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe8dd75d8ff44f5bd62d0c35a67ba0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } }
I20260620 21:49:32.488409 22086 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:32.488497 22086 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe8dd75d8ff44f5bd62d0c35a67ba0a, State: Initialized, Role: FOLLOWER
I20260620 21:49:32.489006 22086 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe8dd75d8ff44f5bd62d0c35a67ba0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } }
I20260620 21:49:32.489135 22086 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:32.489218 22086 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:32.489347 22086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:32.490638 22086 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe8dd75d8ff44f5bd62d0c35a67ba0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } }
I20260620 21:49:32.491010 22086 leader_election.cc:302] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfe8dd75d8ff44f5bd62d0c35a67ba0a; no voters:
I20260620 21:49:32.491369 22086 leader_election.cc:288] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:32.491437 22096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:32.491654 22096 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [term 1 LEADER]: Becoming Leader. State: Replica: dfe8dd75d8ff44f5bd62d0c35a67ba0a, State: Running, Role: LEADER
I20260620 21:49:32.492009 22096 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe8dd75d8ff44f5bd62d0c35a67ba0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } }
I20260620 21:49:32.492704 22086 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:32.493595 22097 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfe8dd75d8ff44f5bd62d0c35a67ba0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe8dd75d8ff44f5bd62d0c35a67ba0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } } }
I20260620 21:49:32.493785 22097 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:32.494117 22097 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfe8dd75d8ff44f5bd62d0c35a67ba0a. Latest consensus state: current_term: 1 leader_uuid: "dfe8dd75d8ff44f5bd62d0c35a67ba0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe8dd75d8ff44f5bd62d0c35a67ba0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } } }
I20260620 21:49:32.494292 22097 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:32.495041 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:32.490370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44073 (local address 127.0.0.1:45183)
0620 21:49:32.490493 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:32.490497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:32.490601 (+ 104us) server_negotiation.cc:407] Connection header received
0620 21:49:32.490759 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:32.490762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:32.490810 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:32.490908 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:32.491179 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:32.491938 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:32.492841 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:32.493149 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:32.493650 (+ 501us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:32.494229 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:32.494232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:32.494235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:32.494248 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:32.494275 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:32.494287 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:32.494793 (+ 506us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:32.494848 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 21:49:32.494887 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 21:49:32.497880 22112 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:49:32.497967 22112 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:49:32.498025 22102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:32.498728 22102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:32.504527 22102 catalog_manager.cc:1347] Generated new cluster ID: 34158dd2b2df40699616c928a7e422b7
I20260620 21:49:32.504612 22102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:32.520304 22102 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:32.521283 22102 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:32.530838 22102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfe8dd75d8ff44f5bd62d0c35a67ba0a: Generated new TSK 0
I20260620 21:49:32.531598 22102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:32.546887 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 22018
W20260620 21:49:32.552495 22003 connection.cc:535] client connection to 127.0.0.1:45183 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45183): Connection reset by peer (error 104)
2026-06-20T21:49:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:32 dist-test-slave-5bvn krb5kdc[22117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:32 dist-test-slave-5bvn krb5kdc[22117](info): set up 2 sockets
Jun 20 21:49:32 dist-test-slave-5bvn krb5kdc[22117](info): commencing operation
krb5kdc: starting...
W20260620 21:49:33.614135 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:33 dist-test-slave-5bvn krb5kdc[22117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:49:33.744721 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:33.858870 22130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:33.859166 22130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:33.859210 22130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:33.863401 22130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:33.863477 22130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:33.863498 22130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:33.863515 22130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:33.863531 22130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:33.869174 22130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39087
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:33.870337 22130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:33.871322 22130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:33.880785 22136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:33.880777 22135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:33.880882 22130 server_base.cc:1034] running on GCE node
W20260620 21:49:33.880777 22138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:33.881564 22130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:33.882800 22130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:33.883992 22130 hybrid_clock.cc:648] HybridClock initialized: now 1781992173883965 us; error 41 us; skew 500 ppm
Jun 20 21:49:33 dist-test-slave-5bvn krb5kdc[22117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:33.887003 22130 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:49:33.887969 22130 webserver.cc:460] Webserver started at http://127.0.0.1:43513/ using document root <none> and password file <none>
I20260620 21:49:33.888489 22130 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:33.888546 22130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:33.888710 22130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:33.890779 22130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "44eaa7cf933d495d828046ff82d1173f"
format_stamp: "Formatted at 2026-06-20 21:49:33 on dist-test-slave-5bvn"
server_key: "4172af14b60eb3452fedd1fd6a838c1d"
server_key_iv: "52dee18a9987243e83af5d9174bbe35f"
server_key_version: "encryptionkey@0"
I20260620 21:49:33.891283 22130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "44eaa7cf933d495d828046ff82d1173f"
format_stamp: "Formatted at 2026-06-20 21:49:33 on dist-test-slave-5bvn"
server_key: "4172af14b60eb3452fedd1fd6a838c1d"
server_key_iv: "52dee18a9987243e83af5d9174bbe35f"
server_key_version: "encryptionkey@0"
I20260620 21:49:33.894899 22130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:49:33.897269 22145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:33.898643 22130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:49:33.898793 22130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "44eaa7cf933d495d828046ff82d1173f"
format_stamp: "Formatted at 2026-06-20 21:49:33 on dist-test-slave-5bvn"
server_key: "4172af14b60eb3452fedd1fd6a838c1d"
server_key_iv: "52dee18a9987243e83af5d9174bbe35f"
server_key_version: "encryptionkey@0"
I20260620 21:49:33.898911 22130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:33.903568 22130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:33.906836 22130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:33.907019 22130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:33.913954 22130 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39087
I20260620 21:49:33.913997 22197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39087 every 8 connection(s)
I20260620 21:49:33.914896 22130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:33.918040 22198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:33.921442 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 22130
I20260620 21:49:33.921607 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:33.921986 1646 external_mini_cluster.cc:1428] Setting key 6b58853e9c24996f05c7fbd740a9a637
I20260620 21:49:33.923985 22198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f: Bootstrap starting.
I20260620 21:49:33.926385 22198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:33.927134 22198 log.cc:824] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:33.927343 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:33.923384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54133 (local address 127.0.0.1:39087)
0620 21:49:33.923856 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:33.923871 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:33.923908 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:49:33.924594 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:33.924611 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:33.924915 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:33.925255 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:33.926078 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:33.926082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:33.926097 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:33.926107 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:33.926159 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:33.926177 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:33.926294 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:33.926416 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 21:49:33.926583 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":147,"threads_started":1}
I20260620 21:49:33.929176 22198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f: No bootstrap required, opened a new log
I20260620 21:49:33.930418 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:33.933380 22198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44eaa7cf933d495d828046ff82d1173f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39087 } }
I20260620 21:49:33.933624 22198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:33.933702 22198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44eaa7cf933d495d828046ff82d1173f, State: Initialized, Role: FOLLOWER
I20260620 21:49:33.934234 22198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44eaa7cf933d495d828046ff82d1173f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39087 } }
I20260620 21:49:33.934500 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:33.932423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42665 (local address 127.0.0.1:39087)
0620 21:49:33.932848 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:33.932852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:33.932865 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:49:33.932897 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:33.932899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:33.932929 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:33.933016 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:33.933861 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:33.933866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:33.933868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:33.933883 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:33.933912 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:33.933930 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:33.934037 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:33.934150 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 21:49:33.934185 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351}
I20260620 21:49:33.934566 22198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:33.934652 22198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:33.934777 22198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:33.935724 22198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44eaa7cf933d495d828046ff82d1173f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39087 } }
I20260620 21:49:33.936076 22198 leader_election.cc:302] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44eaa7cf933d495d828046ff82d1173f; no voters:
I20260620 21:49:33.936393 22198 leader_election.cc:288] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:33.936509 22208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:33.936762 22208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [term 1 LEADER]: Becoming Leader. State: Replica: 44eaa7cf933d495d828046ff82d1173f, State: Running, Role: LEADER
I20260620 21:49:33.937095 22208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44eaa7cf933d495d828046ff82d1173f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39087 } }
I20260620 21:49:33.937664 22198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:33.938971 22209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44eaa7cf933d495d828046ff82d1173f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44eaa7cf933d495d828046ff82d1173f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39087 } } }
I20260620 21:49:33.938988 22210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44eaa7cf933d495d828046ff82d1173f. Latest consensus state: current_term: 1 leader_uuid: "44eaa7cf933d495d828046ff82d1173f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44eaa7cf933d495d828046ff82d1173f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39087 } } }
I20260620 21:49:33.939145 22209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:33.939162 22210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:33.939546 22218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:33.942924 22218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:33.948632 22218 catalog_manager.cc:1347] Generated new cluster ID: 8e9d13640a1c411d9504738e0f797bc2
I20260620 21:49:33.948725 22218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:33.980913 22218 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:33.981298 22218 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:33.988384 22218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44eaa7cf933d495d828046ff82d1173f: Generated new TSK 0
I20260620 21:49:33.989256 22218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:34.005368 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 22130
2026-06-20T21:49:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:34 dist-test-slave-5bvn krb5kdc[22229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:34 dist-test-slave-5bvn krb5kdc[22229](info): set up 2 sockets
Jun 20 21:49:34 dist-test-slave-5bvn krb5kdc[22229](info): commencing operation
krb5kdc: starting...
W20260620 21:49:35.050202 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:35 dist-test-slave-5bvn krb5kdc[22229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:49:35.181777 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:35.294341 22242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:35.294623 22242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:35.294667 22242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:35.298722 22242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:35.298777 22242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:35.298797 22242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:35.298813 22242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:35.298828 22242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:35.303943 22242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:35.304900 22242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:35.305922 22242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:35.314558 22248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:35.314628 22247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:35.314558 22250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:35.314596 22242 server_base.cc:1034] running on GCE node
I20260620 21:49:35.315325 22242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:35.316437 22242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:35.317654 22242 hybrid_clock.cc:648] HybridClock initialized: now 1781992175317610 us; error 48 us; skew 500 ppm
Jun 20 21:49:35 dist-test-slave-5bvn krb5kdc[22229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:35.324667 22242 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:49:35.325649 22242 webserver.cc:460] Webserver started at http://127.0.0.1:46109/ using document root <none> and password file <none>
I20260620 21:49:35.326344 22242 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:35.326426 22242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:35.326606 22242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:35.328832 22242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "af93ecef11c24184829f0d1f2ab0f154"
format_stamp: "Formatted at 2026-06-20 21:49:35 on dist-test-slave-5bvn"
server_key: "f708c36968287194095cb698eaff055a"
server_key_iv: "310020c124582120acafef61e484b896"
server_key_version: "encryptionkey@0"
I20260620 21:49:35.329389 22242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "af93ecef11c24184829f0d1f2ab0f154"
format_stamp: "Formatted at 2026-06-20 21:49:35 on dist-test-slave-5bvn"
server_key: "f708c36968287194095cb698eaff055a"
server_key_iv: "310020c124582120acafef61e484b896"
server_key_version: "encryptionkey@0"
I20260620 21:49:35.333562 22242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 21:49:35.336347 22257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:35.338083 22242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:49:35.338284 22242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "af93ecef11c24184829f0d1f2ab0f154"
format_stamp: "Formatted at 2026-06-20 21:49:35 on dist-test-slave-5bvn"
server_key: "f708c36968287194095cb698eaff055a"
server_key_iv: "310020c124582120acafef61e484b896"
server_key_version: "encryptionkey@0"
I20260620 21:49:35.338410 22242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:35.354561 22242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:35.358026 22242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:35.358253 22242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:35.365582 22242 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38985
I20260620 21:49:35.365604 22309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38985 every 8 connection(s)
I20260620 21:49:35.366575 22242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:35.368675 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 22242
I20260620 21:49:35.368846 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:35.369132 1646 external_mini_cluster.cc:1428] Setting key dd22e94342025bbe23769cb2c0d52f70
I20260620 21:49:35.369467 22310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:35.376030 22310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154: Bootstrap starting.
Jun 20 21:49:35 dist-test-slave-5bvn krb5kdc[22229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 21:49:35.378547 22310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:35.379299 22310 log.cc:824] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:35.381232 22310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154: No bootstrap required, opened a new log
I20260620 21:49:35.383199 22313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:35.370575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44205 (local address 127.0.0.1:38985)
0620 21:49:35.371048 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:35.371063 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:35.371099 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:49:35.371763 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:35.371779 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:35.372116 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:35.372470 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:35.373325 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:35.374411 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:35.375303 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:35.375764 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:35.375937 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:35.378508 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:35.378513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:35.378527 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:35.378558 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:35.380490 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:35.380946 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:35.380951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:35.380955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:35.381058 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:35.381345 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:35.381349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:35.381351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:35.381737 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:35.382002 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:35.382213 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 21:49:35.382510 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":156,"threads_started":1}
I20260620 21:49:35.384980 22310 raft_consensus.cc:348] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af93ecef11c24184829f0d1f2ab0f154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38985 } }
I20260620 21:49:35.385241 22310 raft_consensus.cc:374] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:35.385300 22310 raft_consensus.cc:729] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af93ecef11c24184829f0d1f2ab0f154, State: Initialized, Role: FOLLOWER
I20260620 21:49:35.385860 22310 consensus_queue.cc:260] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af93ecef11c24184829f0d1f2ab0f154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38985 } }
I20260620 21:49:35.386034 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:35.386041 22310 raft_consensus.cc:388] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:35.386108 22310 raft_consensus.cc:482] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:35.386201 22310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:35.387166 22310 raft_consensus.cc:504] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af93ecef11c24184829f0d1f2ab0f154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38985 } }
I20260620 21:49:35.387518 22310 leader_election.cc:302] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af93ecef11c24184829f0d1f2ab0f154; no voters:
I20260620 21:49:35.387863 22310 leader_election.cc:288] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:35.387965 22319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:35.388177 22319 raft_consensus.cc:686] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [term 1 LEADER]: Becoming Leader. State: Replica: af93ecef11c24184829f0d1f2ab0f154, State: Running, Role: LEADER
I20260620 21:49:35.388500 22319 consensus_queue.cc:237] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af93ecef11c24184829f0d1f2ab0f154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38985 } }
I20260620 21:49:35.389211 22310 sys_catalog.cc:564] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:35.389631 22319 sys_catalog.cc:455] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af93ecef11c24184829f0d1f2ab0f154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af93ecef11c24184829f0d1f2ab0f154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38985 } } }
I20260620 21:49:35.389745 22319 sys_catalog.cc:458] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:35.389951 22319 sys_catalog.cc:455] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af93ecef11c24184829f0d1f2ab0f154. Latest consensus state: current_term: 1 leader_uuid: "af93ecef11c24184829f0d1f2ab0f154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af93ecef11c24184829f0d1f2ab0f154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38985 } } }
I20260620 21:49:35.390055 22319 sys_catalog.cc:458] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:35.390396 22324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:35.394539 22324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:35.400388 22313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:35.388086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33795 (local address 127.0.0.1:38985)
0620 21:49:35.388205 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:35.388210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:35.391582 (+ 3372us) server_negotiation.cc:407] Connection header received
0620 21:49:35.391703 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:35.391706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:35.391755 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:35.391849 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:35.392873 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:35.393679 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:35.394579 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:35.394942 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:35.395062 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:35.396576 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:35.396579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:35.396582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:35.396607 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:35.398477 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:35.399026 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:35.399029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:35.399031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:35.399119 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:35.399491 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:35.399495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:35.399496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:35.399697 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:35.399801 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:35.400098 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 21:49:35.400193 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:49:35.407711 22324 catalog_manager.cc:1347] Generated new cluster ID: 4e4393a019534082a0327de8d43e8d62
I20260620 21:49:35.407804 22324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:35.438524 22324 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:35.439584 22324 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:35.445662 22324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af93ecef11c24184829f0d1f2ab0f154: Generated new TSK 0
I20260620 21:49:35.446384 22324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:35.460198 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 22242
2026-06-20T21:49:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:35 dist-test-slave-5bvn krb5kdc[22341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:35 dist-test-slave-5bvn krb5kdc[22341](info): set up 2 sockets
Jun 20 21:49:35 dist-test-slave-5bvn krb5kdc[22341](info): commencing operation
krb5kdc: starting...
W20260620 21:49:36.514292 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:36 dist-test-slave-5bvn krb5kdc[22341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:49:36.639194 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38241
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:36.753456 22354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:36.753754 22354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:36.753798 22354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:36.758100 22354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:36.758175 22354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:36.758195 22354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:36.758213 22354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:36.758229 22354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:36.763707 22354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:36.764789 22354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:36.765944 22354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:36.775601 22359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:36.775588 22360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:36.775588 22362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:36.775632 22354 server_base.cc:1034] running on GCE node
I20260620 21:49:36.776625 22354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:36.777817 22354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:36.779088 22354 hybrid_clock.cc:648] HybridClock initialized: now 1781992176779070 us; error 41 us; skew 500 ppm
Jun 20 21:49:36 dist-test-slave-5bvn krb5kdc[22341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:36.782038 22354 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:49:36.783324 22354 webserver.cc:460] Webserver started at http://127.0.0.1:43761/ using document root <none> and password file <none>
I20260620 21:49:36.783943 22354 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:36.784044 22354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:36.784281 22354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:36.786775 22354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "6aafd0207d264536a81cc6437ac7260f"
format_stamp: "Formatted at 2026-06-20 21:49:36 on dist-test-slave-5bvn"
server_key: "113d0cbdad0278e5c162fb863dbd5b61"
server_key_iv: "35b5e261e21c03334687476a6e5c5214"
server_key_version: "encryptionkey@0"
I20260620 21:49:36.787663 22354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "6aafd0207d264536a81cc6437ac7260f"
format_stamp: "Formatted at 2026-06-20 21:49:36 on dist-test-slave-5bvn"
server_key: "113d0cbdad0278e5c162fb863dbd5b61"
server_key_iv: "35b5e261e21c03334687476a6e5c5214"
server_key_version: "encryptionkey@0"
I20260620 21:49:36.791896 22354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:49:36.794456 22369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:36.795706 22354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:49:36.795874 22354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "6aafd0207d264536a81cc6437ac7260f"
format_stamp: "Formatted at 2026-06-20 21:49:36 on dist-test-slave-5bvn"
server_key: "113d0cbdad0278e5c162fb863dbd5b61"
server_key_iv: "35b5e261e21c03334687476a6e5c5214"
server_key_version: "encryptionkey@0"
I20260620 21:49:36.796000 22354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:36.831700 22354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:36.835062 22354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:36.835299 22354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:36.843061 22354 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38241
I20260620 21:49:36.843077 22421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38241 every 8 connection(s)
I20260620 21:49:36.844087 22354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:36.846868 22422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:36.846894 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 22354
I20260620 21:49:36.847103 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:36.847415 1646 external_mini_cluster.cc:1428] Setting key 3b172697872852cfeb48d1ac1797714b
I20260620 21:49:36.854460 22422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f: Bootstrap starting.
I20260620 21:49:36.857319 22422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:36.858091 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:36.848871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34755 (local address 127.0.0.1:38241)
0620 21:49:36.849330 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:36.849349 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:36.849391 (+ 42us) server_negotiation.cc:407] Connection header received
0620 21:49:36.850419 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:36.850444 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:36.850896 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:36.851274 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:36.851701 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:36.853218 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:36.854227 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:36.854910 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:36.855068 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:36.855939 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:36.855946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:36.855963 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:36.856451 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:36.856533 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:36.856564 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:36.856665 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:36.856893 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 21:49:36.857103 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":128,"threads_started":1}
I20260620 21:49:36.858234 22422 log.cc:824] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:36.860556 22422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f: No bootstrap required, opened a new log
I20260620 21:49:36.861497 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:36.865566 22422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aafd0207d264536a81cc6437ac7260f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38241 } }
I20260620 21:49:36.865841 22422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:36.865923 22422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aafd0207d264536a81cc6437ac7260f, State: Initialized, Role: FOLLOWER
I20260620 21:49:36.866423 22422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aafd0207d264536a81cc6437ac7260f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38241 } }
I20260620 21:49:36.866598 22422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:36.866659 22422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:36.866767 22422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:36.868099 22422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aafd0207d264536a81cc6437ac7260f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38241 } }
I20260620 21:49:36.868213 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:36.863791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42617 (local address 127.0.0.1:38241)
0620 21:49:36.863939 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:36.863944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:36.864061 (+ 117us) server_negotiation.cc:407] Connection header received
0620 21:49:36.864143 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:36.864146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:36.864195 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:36.864286 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:36.864535 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:36.865334 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:36.866345 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:36.866714 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:36.866828 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:36.867731 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:36.867734 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:36.867739 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:36.867758 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:36.867791 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:36.867804 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:36.867886 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:36.867971 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 21:49:36.868038 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:49:36.868512 22422 leader_election.cc:302] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aafd0207d264536a81cc6437ac7260f; no voters:
I20260620 21:49:36.868862 22422 leader_election.cc:288] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:36.868975 22432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:36.869220 22432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [term 1 LEADER]: Becoming Leader. State: Replica: 6aafd0207d264536a81cc6437ac7260f, State: Running, Role: LEADER
I20260620 21:49:36.869578 22432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aafd0207d264536a81cc6437ac7260f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38241 } }
I20260620 21:49:36.870231 22422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:36.870850 22434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aafd0207d264536a81cc6437ac7260f. Latest consensus state: current_term: 1 leader_uuid: "6aafd0207d264536a81cc6437ac7260f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aafd0207d264536a81cc6437ac7260f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38241 } } }
I20260620 21:49:36.871163 22434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:36.871341 22433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aafd0207d264536a81cc6437ac7260f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aafd0207d264536a81cc6437ac7260f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38241 } } }
I20260620 21:49:36.871438 22433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:36.871681 22441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:36.874657 22441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:36.880012 22441 catalog_manager.cc:1347] Generated new cluster ID: f8b0dcc1e6fa44c9892d3960e06aac41
I20260620 21:49:36.880093 22441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:36.885962 22441 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:36.887229 22441 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:36.895645 22441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aafd0207d264536a81cc6437ac7260f: Generated new TSK 0
I20260620 21:49:36.896425 22441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:36.906122 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 22354
2026-06-20T21:49:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:36 dist-test-slave-5bvn krb5kdc[22453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:36 dist-test-slave-5bvn krb5kdc[22453](info): set up 2 sockets
Jun 20 21:49:36 dist-test-slave-5bvn krb5kdc[22453](info): commencing operation
krb5kdc: starting...
W20260620 21:49:37.958587 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:38 dist-test-slave-5bvn krb5kdc[22453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:49:38.085569 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34965
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:38.200419 22466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:38.200688 22466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:38.200730 22466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:38.204838 22466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:38.204912 22466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:38.204932 22466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:38.204948 22466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:38.204964 22466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:38.210501 22466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:38.211607 22466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:38.212651 22466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:38.221715 22471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:38.221697 22472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:38.221715 22474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:38.222054 22466 server_base.cc:1034] running on GCE node
I20260620 21:49:38.222663 22466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:38.223927 22466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:38.225127 22466 hybrid_clock.cc:648] HybridClock initialized: now 1781992178225104 us; error 37 us; skew 500 ppm
Jun 20 21:49:38 dist-test-slave-5bvn krb5kdc[22453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:38.228194 22466 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:49:38.229202 22466 webserver.cc:460] Webserver started at http://127.0.0.1:34083/ using document root <none> and password file <none>
I20260620 21:49:38.229766 22466 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:38.229827 22466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:38.230129 22466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:38.232208 22466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "82598c994290442daf4f1f337e47f9c9"
format_stamp: "Formatted at 2026-06-20 21:49:38 on dist-test-slave-5bvn"
server_key: "540c3ead1e06aa6431c69c9ffa97eb2d"
server_key_iv: "4e254d32342e51b879cf5a3eb2e3e0d4"
server_key_version: "encryptionkey@0"
I20260620 21:49:38.232777 22466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "82598c994290442daf4f1f337e47f9c9"
format_stamp: "Formatted at 2026-06-20 21:49:38 on dist-test-slave-5bvn"
server_key: "540c3ead1e06aa6431c69c9ffa97eb2d"
server_key_iv: "4e254d32342e51b879cf5a3eb2e3e0d4"
server_key_version: "encryptionkey@0"
I20260620 21:49:38.236636 22466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 21:49:38.239415 22481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:38.240775 22466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:49:38.240950 22466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "82598c994290442daf4f1f337e47f9c9"
format_stamp: "Formatted at 2026-06-20 21:49:38 on dist-test-slave-5bvn"
server_key: "540c3ead1e06aa6431c69c9ffa97eb2d"
server_key_iv: "4e254d32342e51b879cf5a3eb2e3e0d4"
server_key_version: "encryptionkey@0"
I20260620 21:49:38.241083 22466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:38.245723 22466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:38.248718 22466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:38.248926 22466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:38.256181 22466 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34965
I20260620 21:49:38.256204 22533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34965 every 8 connection(s)
I20260620 21:49:38.257144 22466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:38.259996 22534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:38.261957 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 22466
I20260620 21:49:38.262100 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:38.262383 1646 external_mini_cluster.cc:1428] Setting key 7e261487342c804e1becb6b5d0bdc107
I20260620 21:49:38.266819 22534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9: Bootstrap starting.
I20260620 21:49:38.267738 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:38.263894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32821 (local address 127.0.0.1:34965)
0620 21:49:38.264341 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:38.264356 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:38.264398 (+ 42us) server_negotiation.cc:407] Connection header received
0620 21:49:38.265089 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:38.265108 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:38.265390 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:38.265725 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:38.266535 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:38.266540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:38.266551 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:38.266562 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:38.266614 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:38.266633 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:38.266780 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:38.266852 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 21:49:38.267026 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":135,"threads_started":1}
I20260620 21:49:38.269268 22534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:38.270226 22534 log.cc:824] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:38.270659 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:38.272380 22534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9: No bootstrap required, opened a new log
I20260620 21:49:38.275480 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:38.272647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50383 (local address 127.0.0.1:34965)
0620 21:49:38.272765 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:38.272768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:38.274127 (+ 1359us) server_negotiation.cc:407] Connection header received
0620 21:49:38.274253 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:38.274255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:38.274287 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:38.274356 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:38.275063 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:38.275066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:38.275068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:38.275078 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:38.275103 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:38.275114 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:38.275182 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:38.275289 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 21:49:38.275322 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 21:49:38.276871 22534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82598c994290442daf4f1f337e47f9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34965 } }
I20260620 21:49:38.277102 22534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:38.277165 22534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82598c994290442daf4f1f337e47f9c9, State: Initialized, Role: FOLLOWER
I20260620 21:49:38.277633 22534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82598c994290442daf4f1f337e47f9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34965 } }
I20260620 21:49:38.277755 22534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:38.277822 22534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:38.277982 22534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:38.278924 22534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82598c994290442daf4f1f337e47f9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34965 } }
I20260620 21:49:38.279278 22534 leader_election.cc:302] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82598c994290442daf4f1f337e47f9c9; no voters:
I20260620 21:49:38.279579 22534 leader_election.cc:288] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:38.279767 22544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:38.280079 22544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [term 1 LEADER]: Becoming Leader. State: Replica: 82598c994290442daf4f1f337e47f9c9, State: Running, Role: LEADER
I20260620 21:49:38.280413 22544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82598c994290442daf4f1f337e47f9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34965 } }
I20260620 21:49:38.280897 22534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:38.282152 22545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82598c994290442daf4f1f337e47f9c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82598c994290442daf4f1f337e47f9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34965 } } }
I20260620 21:49:38.282276 22545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:38.282835 22546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82598c994290442daf4f1f337e47f9c9. Latest consensus state: current_term: 1 leader_uuid: "82598c994290442daf4f1f337e47f9c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82598c994290442daf4f1f337e47f9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34965 } } }
I20260620 21:49:38.282995 22546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9 [sys.catalog]: This master's current role is: LEADER
W20260620 21:49:38.285537 22559 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:49:38.285600 22559 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:49:38.285681 22554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:38.286401 22554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:38.291849 22554 catalog_manager.cc:1347] Generated new cluster ID: 1ff1f8b2f558453da12cd7b21c1e6be7
I20260620 21:49:38.291914 22554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:38.319916 22554 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:38.320271 22554 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:38.340238 22554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82598c994290442daf4f1f337e47f9c9: Generated new TSK 0
I20260620 21:49:38.341048 22554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:38.346493 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 22466
2026-06-20T21:49:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:38 dist-test-slave-5bvn krb5kdc[22565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:38 dist-test-slave-5bvn krb5kdc[22565](info): set up 2 sockets
Jun 20 21:49:38 dist-test-slave-5bvn krb5kdc[22565](info): commencing operation
krb5kdc: starting...
W20260620 21:49:39.394145 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:39 dist-test-slave-5bvn krb5kdc[22565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:49:39.517480 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35125
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:39.626773 22578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:39.627039 22578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:39.627081 22578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:39.631059 22578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:39.631124 22578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:39.631143 22578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:39.631160 22578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:39.631175 22578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:39.636530 22578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:39.637548 22578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:39.638522 22578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:39.646898 22584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:39.646967 22578 server_base.cc:1034] running on GCE node
W20260620 21:49:39.646917 22583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:39.646898 22586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:39.647779 22578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:39.648861 22578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:39.650061 22578 hybrid_clock.cc:648] HybridClock initialized: now 1781992179650027 us; error 46 us; skew 500 ppm
Jun 20 21:49:39 dist-test-slave-5bvn krb5kdc[22565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:39.652900 22578 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:49:39.653914 22578 webserver.cc:460] Webserver started at http://127.0.0.1:34599/ using document root <none> and password file <none>
I20260620 21:49:39.654492 22578 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:39.654582 22578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:39.654803 22578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:39.656934 22578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "3977141abf074a51b560081dc2e40373"
format_stamp: "Formatted at 2026-06-20 21:49:39 on dist-test-slave-5bvn"
server_key: "f5fdfb40392215cd25c37c0fe231edbc"
server_key_iv: "8d485a0b525bd9f65169f1be167385a0"
server_key_version: "encryptionkey@0"
I20260620 21:49:39.657523 22578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "3977141abf074a51b560081dc2e40373"
format_stamp: "Formatted at 2026-06-20 21:49:39 on dist-test-slave-5bvn"
server_key: "f5fdfb40392215cd25c37c0fe231edbc"
server_key_iv: "8d485a0b525bd9f65169f1be167385a0"
server_key_version: "encryptionkey@0"
I20260620 21:49:39.661602 22578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:49:39.664247 22593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:39.665481 22578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:49:39.665657 22578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "3977141abf074a51b560081dc2e40373"
format_stamp: "Formatted at 2026-06-20 21:49:39 on dist-test-slave-5bvn"
server_key: "f5fdfb40392215cd25c37c0fe231edbc"
server_key_iv: "8d485a0b525bd9f65169f1be167385a0"
server_key_version: "encryptionkey@0"
I20260620 21:49:39.665781 22578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:39.705957 22578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:39.709609 22578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:39.709826 22578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:39.716866 22578 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35125
I20260620 21:49:39.716890 22645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35125 every 8 connection(s)
I20260620 21:49:39.717821 22578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:39.720489 22646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:39.724287 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 22578
I20260620 21:49:39.724470 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:39.724779 1646 external_mini_cluster.cc:1428] Setting key dfd7d16a13083fe70fe95625c81bc796
I20260620 21:49:39.726461 22646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373: Bootstrap starting.
I20260620 21:49:39.728619 22646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:39.729347 22646 log.cc:824] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:39.731226 22646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373: No bootstrap required, opened a new log
Jun 20 21:49:39 dist-test-slave-5bvn krb5kdc[22565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 21:49:39.735237 22646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3977141abf074a51b560081dc2e40373" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35125 } }
I20260620 21:49:39.735545 22646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:39.735666 22646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3977141abf074a51b560081dc2e40373, State: Initialized, Role: FOLLOWER
I20260620 21:49:39.736217 22646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3977141abf074a51b560081dc2e40373" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35125 } }
I20260620 21:49:39.736353 22646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:39.736446 22646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:39.736575 22646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:39.737509 22646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3977141abf074a51b560081dc2e40373" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35125 } }
I20260620 21:49:39.737921 22646 leader_election.cc:302] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3977141abf074a51b560081dc2e40373; no voters:
I20260620 21:49:39.738263 22646 leader_election.cc:288] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:39.738387 22651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:39.738618 22651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [term 1 LEADER]: Becoming Leader. State: Replica: 3977141abf074a51b560081dc2e40373, State: Running, Role: LEADER
I20260620 21:49:39.738942 22651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3977141abf074a51b560081dc2e40373" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35125 } }
I20260620 21:49:39.739751 22646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:39.740223 22651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3977141abf074a51b560081dc2e40373. Latest consensus state: current_term: 1 leader_uuid: "3977141abf074a51b560081dc2e40373" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3977141abf074a51b560081dc2e40373" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35125 } } }
I20260620 21:49:39.740337 22651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:39.741112 22652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3977141abf074a51b560081dc2e40373" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3977141abf074a51b560081dc2e40373" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35125 } } }
I20260620 21:49:39.741161 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:39.726255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:48113 (local address 127.0.0.1:35125)
0620 21:49:39.726775 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:39.726790 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:39.726828 (+ 38us) server_negotiation.cc:407] Connection header received
0620 21:49:39.727467 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:39.727483 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:39.727852 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:39.728205 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:39.729061 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:39.730159 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:39.731032 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:39.731546 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:39.731711 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:39.734871 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:39.734890 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:39.734906 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:39.734935 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:39.736997 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:39.737568 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:39.737575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:39.737583 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:39.737732 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:39.738135 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:39.738138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:39.738140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:39.738721 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:39.739539 (+ 818us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:39.739714 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 21:49:39.739954 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":135,"threads_started":1}
I20260620 21:49:39.741216 22652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:39.741606 22655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:39.744529 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:39.744848 22655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:39.751503 22655 catalog_manager.cc:1347] Generated new cluster ID: be69b3c82bd34c8e89847b96a8a759d0
I20260620 21:49:39.751638 22655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:39.755705 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:39.746744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:36841 (local address 127.0.0.1:35125)
0620 21:49:39.746862 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:39.746868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:39.747214 (+ 346us) server_negotiation.cc:407] Connection header received
0620 21:49:39.747371 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:39.747374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:39.747424 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:39.747515 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:39.748165 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:39.748692 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:39.749702 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:39.750082 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:39.750187 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:39.752053 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:39.752056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:39.752059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:39.752084 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:39.753768 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:39.754379 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:39.754382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:39.754385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:39.754469 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:39.754818 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:39.754821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:39.754823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:39.755015 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:39.755139 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:39.755437 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 21:49:39.755543 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 21:49:39.765787 22655 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:39.767256 22655 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:39.776736 22655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3977141abf074a51b560081dc2e40373: Generated new TSK 0
I20260620 21:49:39.777628 22655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:39.784581 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 22578
W20260620 21:49:39.790132 22563 connection.cc:535] client connection to 127.0.0.1:35125 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35125): Connection reset by peer (error 104)
2026-06-20T21:49:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:39 dist-test-slave-5bvn krb5kdc[22677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:39 dist-test-slave-5bvn krb5kdc[22677](info): set up 2 sockets
Jun 20 21:49:39 dist-test-slave-5bvn krb5kdc[22677](info): commencing operation
krb5kdc: starting...
W20260620 21:49:40.842192 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:40 dist-test-slave-5bvn krb5kdc[22677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:49:40.969177 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:41.078989 22690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:41.079310 22690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:41.079392 22690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:41.083444 22690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:41.083540 22690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:41.083576 22690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:41.083640 22690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:41.083676 22690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:41.089149 22690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:41.090240 22690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:41.091261 22690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:41.099581 22698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:41.099581 22695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:41.099663 22696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:41.099846 22690 server_base.cc:1034] running on GCE node
I20260620 21:49:41.100518 22690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:41.101572 22690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:41.102766 22690 hybrid_clock.cc:648] HybridClock initialized: now 1781992181102744 us; error 31 us; skew 500 ppm
Jun 20 21:49:41 dist-test-slave-5bvn krb5kdc[22677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:41.110910 22690 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:49:41.111886 22690 webserver.cc:460] Webserver started at http://127.0.0.1:44499/ using document root <none> and password file <none>
I20260620 21:49:41.112402 22690 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:41.112460 22690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:41.112622 22690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:41.114725 22690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "83010b250ea5409fadfa9c1370083268"
format_stamp: "Formatted at 2026-06-20 21:49:41 on dist-test-slave-5bvn"
server_key: "7ca9d0e36d5a70024b001f0dfc4acc86"
server_key_iv: "919df9fe4a2510396ac140c968ed4ee2"
server_key_version: "encryptionkey@0"
I20260620 21:49:41.115306 22690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "83010b250ea5409fadfa9c1370083268"
format_stamp: "Formatted at 2026-06-20 21:49:41 on dist-test-slave-5bvn"
server_key: "7ca9d0e36d5a70024b001f0dfc4acc86"
server_key_iv: "919df9fe4a2510396ac140c968ed4ee2"
server_key_version: "encryptionkey@0"
I20260620 21:49:41.119134 22690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 21:49:41.122215 22705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:41.123687 22690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:49:41.123859 22690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "83010b250ea5409fadfa9c1370083268"
format_stamp: "Formatted at 2026-06-20 21:49:41 on dist-test-slave-5bvn"
server_key: "7ca9d0e36d5a70024b001f0dfc4acc86"
server_key_iv: "919df9fe4a2510396ac140c968ed4ee2"
server_key_version: "encryptionkey@0"
I20260620 21:49:41.124002 22690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:41.147501 22690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:41.151055 22690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:41.151314 22690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:41.158697 22690 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35665
I20260620 21:49:41.158715 22757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35665 every 8 connection(s)
I20260620 21:49:41.159871 22690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:41.162868 22758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:41.166136 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 22690
I20260620 21:49:41.166272 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:41.166566 1646 external_mini_cluster.cc:1428] Setting key 5683fac947705a28612a3527d660e6ac
I20260620 21:49:41.168774 22758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268: Bootstrap starting.
I20260620 21:49:41.170964 22758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:41.171700 22758 log.cc:824] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:41.173491 22758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268: No bootstrap required, opened a new log
I20260620 21:49:41.175745 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:41.168065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:34855 (local address 127.0.0.1:35665)
0620 21:49:41.168499 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:41.168514 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:41.168548 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:49:41.169231 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:41.169247 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:41.169580 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:41.169956 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:41.170356 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:41.171399 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:41.172329 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:41.172803 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:41.172981 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:41.173924 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:41.173930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:41.173942 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:41.174315 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:41.174367 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:41.174389 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:41.174532 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:41.174704 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 21:49:41.174842 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":136,"threads_started":1}
I20260620 21:49:41.176901 22758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83010b250ea5409fadfa9c1370083268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260620 21:49:41.177153 22758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:41.177229 22758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83010b250ea5409fadfa9c1370083268, State: Initialized, Role: FOLLOWER
I20260620 21:49:41.177747 22758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83010b250ea5409fadfa9c1370083268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260620 21:49:41.177919 22758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:41.177996 22758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:41.178100 22758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:41.178923 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:41.179214 22758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83010b250ea5409fadfa9c1370083268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260620 21:49:41.179549 22758 leader_election.cc:302] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83010b250ea5409fadfa9c1370083268; no voters:
I20260620 21:49:41.179876 22758 leader_election.cc:288] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:41.179944 22763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:41.180167 22763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [term 1 LEADER]: Becoming Leader. State: Replica: 83010b250ea5409fadfa9c1370083268, State: Running, Role: LEADER
I20260620 21:49:41.180470 22763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83010b250ea5409fadfa9c1370083268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260620 21:49:41.181207 22758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:41.181636 22763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83010b250ea5409fadfa9c1370083268. Latest consensus state: current_term: 1 leader_uuid: "83010b250ea5409fadfa9c1370083268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83010b250ea5409fadfa9c1370083268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } } }
I20260620 21:49:41.181743 22763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:41.181970 22763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83010b250ea5409fadfa9c1370083268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83010b250ea5409fadfa9c1370083268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } } }
I20260620 21:49:41.182076 22763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:41.182365 22772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:41.185698 22772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:41.186458 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:41.181128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:42877 (local address 127.0.0.1:35665)
0620 21:49:41.181234 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:41.181239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:41.181548 (+ 309us) server_negotiation.cc:407] Connection header received
0620 21:49:41.181738 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:41.181742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:41.181793 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:41.181870 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:41.182521 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:41.183267 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:41.184457 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:41.184705 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:41.184824 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:41.185942 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:41.185946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:41.185950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:41.185965 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:41.185995 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:41.186010 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:41.186110 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:41.186303 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 21:49:41.186342 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 21:49:41.191606 22772 catalog_manager.cc:1347] Generated new cluster ID: 5b1655f6a9674a20bb1c2c6f9d45f417
I20260620 21:49:41.191673 22772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:41.205986 22772 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:41.206986 22772 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:41.217146 22772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83010b250ea5409fadfa9c1370083268: Generated new TSK 0
I20260620 21:49:41.217942 22772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:41.224987 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 22690
W20260620 21:49:41.230370 22675 connection.cc:535] client connection to 127.0.0.1:35665 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35665): Connection reset by peer (error 104)
2026-06-20T21:49:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:41 dist-test-slave-5bvn krb5kdc[22789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:41 dist-test-slave-5bvn krb5kdc[22789](info): set up 2 sockets
Jun 20 21:49:41 dist-test-slave-5bvn krb5kdc[22789](info): commencing operation
krb5kdc: starting...
W20260620 21:49:42.290292 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:42 dist-test-slave-5bvn krb5kdc[22789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:49:42.418352 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:42.534080 22802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:42.534551 22802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:42.534668 22802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:42.538815 22802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:42.538885 22802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:42.538908 22802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:42.538928 22802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:42.538975 22802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:42.544616 22802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:42.545742 22802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:42.546852 22802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:42.555485 22807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:42.555474 22810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:42.555490 22808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:42.556083 22802 server_base.cc:1034] running on GCE node
I20260620 21:49:42.556630 22802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:42.557749 22802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:42.558924 22802 hybrid_clock.cc:648] HybridClock initialized: now 1781992182558903 us; error 34 us; skew 500 ppm
Jun 20 21:49:42 dist-test-slave-5bvn krb5kdc[22789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:42.561867 22802 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:49:42.562774 22802 webserver.cc:460] Webserver started at http://127.0.0.1:33465/ using document root <none> and password file <none>
I20260620 21:49:42.563302 22802 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:42.563362 22802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:42.563539 22802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:42.565541 22802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "3db2aa01bbdd4c2397e6e36d0ba355c1"
format_stamp: "Formatted at 2026-06-20 21:49:42 on dist-test-slave-5bvn"
server_key: "2ac7e801b9f2c21ed957306422648c2b"
server_key_iv: "19246ad2b2e47e4c394fa8bc8dccf906"
server_key_version: "encryptionkey@0"
I20260620 21:49:42.566150 22802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "3db2aa01bbdd4c2397e6e36d0ba355c1"
format_stamp: "Formatted at 2026-06-20 21:49:42 on dist-test-slave-5bvn"
server_key: "2ac7e801b9f2c21ed957306422648c2b"
server_key_iv: "19246ad2b2e47e4c394fa8bc8dccf906"
server_key_version: "encryptionkey@0"
I20260620 21:49:42.569988 22802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:49:42.572393 22817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:42.573705 22802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:49:42.573879 22802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "3db2aa01bbdd4c2397e6e36d0ba355c1"
format_stamp: "Formatted at 2026-06-20 21:49:42 on dist-test-slave-5bvn"
server_key: "2ac7e801b9f2c21ed957306422648c2b"
server_key_iv: "19246ad2b2e47e4c394fa8bc8dccf906"
server_key_version: "encryptionkey@0"
I20260620 21:49:42.574018 22802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:42.578471 22802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:42.581733 22802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:42.581986 22802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:42.588637 22802 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45257
I20260620 21:49:42.588713 22869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45257 every 8 connection(s)
I20260620 21:49:42.589572 22802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:42.592404 22870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:42.595325 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 22802
I20260620 21:49:42.595495 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:42.595810 1646 external_mini_cluster.cc:1428] Setting key 00edc22b93d8e834f37d1a4e084ea601
I20260620 21:49:42.598143 22870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1: Bootstrap starting.
I20260620 21:49:42.600638 22870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:42.601084 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:42.597129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:42949 (local address 127.0.0.1:45257)
0620 21:49:42.597550 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:42.597567 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:42.597602 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:49:42.598329 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:42.598347 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:42.598645 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:42.598924 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:42.599791 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:42.599796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:42.599807 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:42.599819 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:42.599875 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:42.599896 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:42.600013 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:42.600128 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 21:49:42.600309 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":130,"threads_started":1}
I20260620 21:49:42.601485 22870 log.cc:824] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:42.603595 22870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1: No bootstrap required, opened a new log
I20260620 21:49:42.603940 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:42.607733 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:42.605920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:48023 (local address 127.0.0.1:45257)
0620 21:49:42.606029 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:42.606032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:42.606234 (+ 202us) server_negotiation.cc:407] Connection header received
0620 21:49:42.606359 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:42.606361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:42.606390 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:42.606449 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:42.607225 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:42.607228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:42.607230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:42.607241 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:42.607267 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:42.607277 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:42.607355 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:42.607458 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 21:49:42.607566 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:49:42.608103 22870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2aa01bbdd4c2397e6e36d0ba355c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260620 21:49:42.608352 22870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:42.608430 22870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db2aa01bbdd4c2397e6e36d0ba355c1, State: Initialized, Role: FOLLOWER
I20260620 21:49:42.608887 22870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2aa01bbdd4c2397e6e36d0ba355c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260620 21:49:42.609046 22870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:42.609134 22870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:42.609232 22870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:42.610232 22870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2aa01bbdd4c2397e6e36d0ba355c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260620 21:49:42.610571 22870 leader_election.cc:302] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3db2aa01bbdd4c2397e6e36d0ba355c1; no voters:
I20260620 21:49:42.610884 22870 leader_election.cc:288] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:42.611065 22880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:42.611342 22880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [term 1 LEADER]: Becoming Leader. State: Replica: 3db2aa01bbdd4c2397e6e36d0ba355c1, State: Running, Role: LEADER
I20260620 21:49:42.611698 22880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2aa01bbdd4c2397e6e36d0ba355c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260620 21:49:42.612147 22870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:42.612826 22880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3db2aa01bbdd4c2397e6e36d0ba355c1. Latest consensus state: current_term: 1 leader_uuid: "3db2aa01bbdd4c2397e6e36d0ba355c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2aa01bbdd4c2397e6e36d0ba355c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } } }
I20260620 21:49:42.613018 22880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:42.613245 22881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3db2aa01bbdd4c2397e6e36d0ba355c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2aa01bbdd4c2397e6e36d0ba355c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } } }
I20260620 21:49:42.613368 22881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:42.613582 22889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:42.616858 22889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:42.622591 22889 catalog_manager.cc:1347] Generated new cluster ID: 120b87dade894ec0bee86cd1852aa117
I20260620 21:49:42.622677 22889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:42.639264 22889 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:42.639631 22889 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:42.648144 22889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3db2aa01bbdd4c2397e6e36d0ba355c1: Generated new TSK 0
I20260620 21:49:42.648862 22889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:42.653545 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 22802
2026-06-20T21:49:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:42 dist-test-slave-5bvn krb5kdc[22901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:42 dist-test-slave-5bvn krb5kdc[22901](info): set up 2 sockets
Jun 20 21:49:42 dist-test-slave-5bvn krb5kdc[22901](info): commencing operation
krb5kdc: starting...
W20260620 21:49:43.710256 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:43 dist-test-slave-5bvn krb5kdc[22901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:49:43.836442 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:43.946374 22914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:43.946689 22914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:43.946784 22914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:43.951083 22914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:43.951210 22914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:43.951251 22914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:43.951290 22914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:43.951324 22914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:43.957134 22914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34403
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:43.958328 22914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:43.959364 22914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:43.968079 22922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:43.968111 22919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:43.968087 22920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:43.968552 22914 server_base.cc:1034] running on GCE node
I20260620 21:49:43.969135 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:43.970337 22914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:43.971551 22914 hybrid_clock.cc:648] HybridClock initialized: now 1781992183971526 us; error 44 us; skew 500 ppm
Jun 20 21:49:43 dist-test-slave-5bvn krb5kdc[22901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:43.974483 22914 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:49:43.975320 22914 webserver.cc:460] Webserver started at http://127.0.0.1:44505/ using document root <none> and password file <none>
I20260620 21:49:43.975843 22914 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:43.975903 22914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:43.976064 22914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:43.978102 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "adf4537dfd934013aec08e41cd338c81"
format_stamp: "Formatted at 2026-06-20 21:49:43 on dist-test-slave-5bvn"
server_key: "5be55b4faed3d88599b63a56441da978"
server_key_iv: "730c3f942fa4f1c16fa38bdcbf4d91b5"
server_key_version: "encryptionkey@0"
I20260620 21:49:43.978608 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "adf4537dfd934013aec08e41cd338c81"
format_stamp: "Formatted at 2026-06-20 21:49:43 on dist-test-slave-5bvn"
server_key: "5be55b4faed3d88599b63a56441da978"
server_key_iv: "730c3f942fa4f1c16fa38bdcbf4d91b5"
server_key_version: "encryptionkey@0"
I20260620 21:49:43.982290 22914 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:49:43.984628 22929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:43.985988 22914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:49:43.986119 22914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "adf4537dfd934013aec08e41cd338c81"
format_stamp: "Formatted at 2026-06-20 21:49:43 on dist-test-slave-5bvn"
server_key: "5be55b4faed3d88599b63a56441da978"
server_key_iv: "730c3f942fa4f1c16fa38bdcbf4d91b5"
server_key_version: "encryptionkey@0"
I20260620 21:49:43.986224 22914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:43.997480 22914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:44.000726 22914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:44.000926 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:44.008015 22914 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34403
I20260620 21:49:44.008039 22981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34403 every 8 connection(s)
I20260620 21:49:44.008929 22914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:44.011610 22982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:44.012924 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 22914
I20260620 21:49:44.013074 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:44.013365 1646 external_mini_cluster.cc:1428] Setting key 71cf716584f9f2afb39c107c6e378352
I20260620 21:49:44.017760 22982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81: Bootstrap starting.
I20260620 21:49:44.020375 22982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:44.021152 22982 log.cc:824] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:49:44 dist-test-slave-5bvn krb5kdc[22901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 21:49:44.023339 22982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81: No bootstrap required, opened a new log
I20260620 21:49:44.028352 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:44.014828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:36041 (local address 127.0.0.1:34403)
0620 21:49:44.015268 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:44.015283 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:44.015318 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:49:44.015967 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:44.015985 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:44.016524 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:44.016885 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:44.017681 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:44.018738 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:44.019617 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:44.020099 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:44.020281 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:44.022979 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:44.022986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:44.023002 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:44.023033 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:44.025165 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:44.025717 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:44.025724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:44.025730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:44.025886 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:44.026202 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:44.026206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:44.026208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:44.026716 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:44.026937 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:44.027150 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 21:49:44.027384 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":141,"threads_started":1}
I20260620 21:49:44.028193 22982 raft_consensus.cc:348] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4537dfd934013aec08e41cd338c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260620 21:49:44.028457 22982 raft_consensus.cc:374] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:44.028501 22982 raft_consensus.cc:729] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf4537dfd934013aec08e41cd338c81, State: Initialized, Role: FOLLOWER
I20260620 21:49:44.028949 22982 consensus_queue.cc:260] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4537dfd934013aec08e41cd338c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260620 21:49:44.029110 22982 raft_consensus.cc:388] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:44.029172 22982 raft_consensus.cc:482] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:44.029264 22982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:44.030458 22982 raft_consensus.cc:504] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4537dfd934013aec08e41cd338c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260620 21:49:44.030794 22982 leader_election.cc:302] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adf4537dfd934013aec08e41cd338c81; no voters:
I20260620 21:49:44.031153 22982 leader_election.cc:288] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:44.031270 22987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:44.031498 22987 raft_consensus.cc:686] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [term 1 LEADER]: Becoming Leader. State: Replica: adf4537dfd934013aec08e41cd338c81, State: Running, Role: LEADER
I20260620 21:49:44.031546 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:44.031832 22987 consensus_queue.cc:237] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4537dfd934013aec08e41cd338c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260620 21:49:44.032380 22982 sys_catalog.cc:564] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:44.033066 22987 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adf4537dfd934013aec08e41cd338c81. Latest consensus state: current_term: 1 leader_uuid: "adf4537dfd934013aec08e41cd338c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4537dfd934013aec08e41cd338c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } } }
I20260620 21:49:44.033186 22987 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:44.033679 22989 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adf4537dfd934013aec08e41cd338c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4537dfd934013aec08e41cd338c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } } }
I20260620 21:49:44.033784 22989 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81 [sys.catalog]: This master's current role is: LEADER
W20260620 21:49:44.037797 23007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:49:44.037928 23007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:49:44.038350 23001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:44.039464 23001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:44.042891 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:44.034025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:49339 (local address 127.0.0.1:34403)
0620 21:49:44.034124 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:44.034128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:44.034181 (+ 53us) server_negotiation.cc:407] Connection header received
0620 21:49:44.034385 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:44.034388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:44.034434 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:44.034482 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:44.035519 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:44.036034 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:44.037140 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:44.037484 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:44.037602 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:44.039303 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:44.039308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:44.039310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:44.039335 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:44.040922 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:44.041469 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:44.041474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:44.041476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:44.041559 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:44.041949 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:44.041953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:44.041954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:44.042137 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:44.042290 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:44.042564 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 21:49:44.042705 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 21:49:44.045295 23001 catalog_manager.cc:1347] Generated new cluster ID: c39c76f7a05d4d47b3d876f16c196334
I20260620 21:49:44.045362 23001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:44.064917 23001 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:44.065989 23001 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:44.073884 23001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adf4537dfd934013aec08e41cd338c81: Generated new TSK 0
I20260620 21:49:44.074641 23001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:44.080871 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 22914
W20260620 21:49:44.086079 22899 connection.cc:535] client connection to 127.0.0.1:34403 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34403): Connection reset by peer (error 104)
2026-06-20T21:49:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:44 dist-test-slave-5bvn krb5kdc[23013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:44 dist-test-slave-5bvn krb5kdc[23013](info): set up 2 sockets
Jun 20 21:49:44 dist-test-slave-5bvn krb5kdc[23013](info): commencing operation
krb5kdc: starting...
W20260620 21:49:45.130052 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:45 dist-test-slave-5bvn krb5kdc[23013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:49:45.265406 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:45.375918 23026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:45.376286 23026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:45.376358 23026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:45.380990 23026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:45.381108 23026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:45.381150 23026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:45.381209 23026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:45.381250 23026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:45.387138 23026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44253
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:45.388355 23026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:45.389410 23026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:45.398376 23034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:45.398422 23031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:45.398422 23032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:45.398674 23026 server_base.cc:1034] running on GCE node
I20260620 21:49:45.399638 23026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:45.400826 23026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:45.402011 23026 hybrid_clock.cc:648] HybridClock initialized: now 1781992185401994 us; error 38 us; skew 500 ppm
Jun 20 21:49:45 dist-test-slave-5bvn krb5kdc[23013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:45.404855 23026 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:49:45.405807 23026 webserver.cc:460] Webserver started at http://127.0.0.1:37817/ using document root <none> and password file <none>
I20260620 21:49:45.406417 23026 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:45.406509 23026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:45.406738 23026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:45.408732 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "3496b738555b4188bec2520ba2c08a0e"
format_stamp: "Formatted at 2026-06-20 21:49:45 on dist-test-slave-5bvn"
server_key: "e1fe59db6edea7ac7c681bb6c12dac9b"
server_key_iv: "ba8a5e134756ed29fadf95be6dbcdb85"
server_key_version: "encryptionkey@0"
I20260620 21:49:45.409292 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "3496b738555b4188bec2520ba2c08a0e"
format_stamp: "Formatted at 2026-06-20 21:49:45 on dist-test-slave-5bvn"
server_key: "e1fe59db6edea7ac7c681bb6c12dac9b"
server_key_iv: "ba8a5e134756ed29fadf95be6dbcdb85"
server_key_version: "encryptionkey@0"
I20260620 21:49:45.413053 23026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:49:45.415506 23041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:45.416721 23026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:49:45.416867 23026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "3496b738555b4188bec2520ba2c08a0e"
format_stamp: "Formatted at 2026-06-20 21:49:45 on dist-test-slave-5bvn"
server_key: "e1fe59db6edea7ac7c681bb6c12dac9b"
server_key_iv: "ba8a5e134756ed29fadf95be6dbcdb85"
server_key_version: "encryptionkey@0"
I20260620 21:49:45.416987 23026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:45.428221 23026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:45.431572 23026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:45.431789 23026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:45.439419 23026 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44253
I20260620 21:49:45.439435 23093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44253 every 8 connection(s)
I20260620 21:49:45.440395 23026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:45.441895 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 23026
I20260620 21:49:45.442061 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:45.442368 1646 external_mini_cluster.cc:1428] Setting key cbd473f144f48d865642319ceb0786b1
I20260620 21:49:45.443718 23094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:45.450044 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e: Bootstrap starting.
I20260620 21:49:45.451503 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:45.443701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:35439 (local address 127.0.0.1:44253)
0620 21:49:45.444208 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:45.444224 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:45.444257 (+ 33us) server_negotiation.cc:407] Connection header received
0620 21:49:45.444878 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:45.444894 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:45.445217 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:45.445505 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:45.445816 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:45.446862 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:45.448079 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:45.448557 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:45.448687 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:45.449637 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:45.449642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:45.449655 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:45.450138 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:45.450229 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:45.450269 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:45.450383 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:45.450579 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 21:49:45.450785 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":156,"threads_started":1}
I20260620 21:49:45.452468 23094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:45.453313 23094 log.cc:824] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:45.454696 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:45.455507 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e: No bootstrap required, opened a new log
I20260620 21:49:45.460058 23094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3496b738555b4188bec2520ba2c08a0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44253 } }
I20260620 21:49:45.460378 23094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:45.460472 23094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3496b738555b4188bec2520ba2c08a0e, State: Initialized, Role: FOLLOWER
I20260620 21:49:45.461021 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:45.456728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:40331 (local address 127.0.0.1:44253)
0620 21:49:45.456856 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:45.456860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:45.457017 (+ 157us) server_negotiation.cc:407] Connection header received
0620 21:49:45.457147 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:45.457149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:45.457186 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:45.457249 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:45.457550 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:45.458105 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:45.459278 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:45.459500 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:45.459581 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:45.460452 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:45.460456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:45.460459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:45.460478 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:45.460522 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:45.460534 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:45.460628 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:45.460818 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 21:49:45.460859 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:49:45.460973 23094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3496b738555b4188bec2520ba2c08a0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44253 } }
I20260620 21:49:45.461144 23094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:45.461218 23094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:45.461313 23094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:45.462332 23094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3496b738555b4188bec2520ba2c08a0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44253 } }
I20260620 21:49:45.462663 23094 leader_election.cc:302] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3496b738555b4188bec2520ba2c08a0e; no voters:
I20260620 21:49:45.462996 23094 leader_election.cc:288] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:45.463083 23104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:45.463315 23104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [term 1 LEADER]: Becoming Leader. State: Replica: 3496b738555b4188bec2520ba2c08a0e, State: Running, Role: LEADER
I20260620 21:49:45.463606 23104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3496b738555b4188bec2520ba2c08a0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44253 } }
I20260620 21:49:45.464190 23094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:45.465216 23106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3496b738555b4188bec2520ba2c08a0e. Latest consensus state: current_term: 1 leader_uuid: "3496b738555b4188bec2520ba2c08a0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3496b738555b4188bec2520ba2c08a0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44253 } } }
I20260620 21:49:45.465348 23106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:45.465619 23105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3496b738555b4188bec2520ba2c08a0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3496b738555b4188bec2520ba2c08a0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44253 } } }
I20260620 21:49:45.465706 23105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:45.465806 23113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:45.468566 23113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:45.474242 23113 catalog_manager.cc:1347] Generated new cluster ID: 5b450d78a05d4e23a4b968daf3a245a0
I20260620 21:49:45.474319 23113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:45.489341 23113 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:45.490625 23113 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:45.507848 23113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3496b738555b4188bec2520ba2c08a0e: Generated new TSK 0
I20260620 21:49:45.508816 23113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:45.519472 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 23026
W20260620 21:49:45.524828 23011 connection.cc:535] client connection to 127.0.0.1:44253 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44253): Connection reset by peer (error 104)
2026-06-20T21:49:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:45 dist-test-slave-5bvn krb5kdc[23125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:45 dist-test-slave-5bvn krb5kdc[23125](info): set up 2 sockets
Jun 20 21:49:45 dist-test-slave-5bvn krb5kdc[23125](info): commencing operation
krb5kdc: starting...
W20260620 21:49:46.574174 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:46 dist-test-slave-5bvn krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:49:46.696879 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41461
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:46.805641 23138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:46.805997 23138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:46.806046 23138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:46.810060 23138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:46.810122 23138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:46.810146 23138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:46.810163 23138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:46.810178 23138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:46.815505 23138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41461
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:46.816529 23138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:46.817514 23138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:46.826112 23144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:46.826290 23146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:46.826215 23143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:46.826320 23138 server_base.cc:1034] running on GCE node
I20260620 21:49:46.826918 23138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:46.828092 23138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:46.829272 23138 hybrid_clock.cc:648] HybridClock initialized: now 1781992186829242 us; error 43 us; skew 500 ppm
Jun 20 21:49:46 dist-test-slave-5bvn krb5kdc[23125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:46.837677 23138 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:49:46.838821 23138 webserver.cc:460] Webserver started at http://127.0.0.1:34595/ using document root <none> and password file <none>
I20260620 21:49:46.839480 23138 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:46.839588 23138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:46.839851 23138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:46.842435 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "10293f3c7ca34856aa3226fde913213f"
format_stamp: "Formatted at 2026-06-20 21:49:46 on dist-test-slave-5bvn"
server_key: "b9a7a414cfbc3d179f922adb3d861e41"
server_key_iv: "4ec9fc85fc1917bdb6c9edc61ea4a12a"
server_key_version: "encryptionkey@0"
I20260620 21:49:46.843061 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "10293f3c7ca34856aa3226fde913213f"
format_stamp: "Formatted at 2026-06-20 21:49:46 on dist-test-slave-5bvn"
server_key: "b9a7a414cfbc3d179f922adb3d861e41"
server_key_iv: "4ec9fc85fc1917bdb6c9edc61ea4a12a"
server_key_version: "encryptionkey@0"
I20260620 21:49:46.847012 23138 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:49:46.850425 23153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:46.852208 23138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:49:46.852416 23138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "10293f3c7ca34856aa3226fde913213f"
format_stamp: "Formatted at 2026-06-20 21:49:46 on dist-test-slave-5bvn"
server_key: "b9a7a414cfbc3d179f922adb3d861e41"
server_key_iv: "4ec9fc85fc1917bdb6c9edc61ea4a12a"
server_key_version: "encryptionkey@0"
I20260620 21:49:46.852559 23138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:46.857442 23138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:46.860584 23138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:46.860843 23138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:46.867995 23138 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41461
I20260620 21:49:46.868024 23205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41461 every 8 connection(s)
I20260620 21:49:46.869051 23138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:46.871968 23206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:46.873216 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 23138
I20260620 21:49:46.873333 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:46.873668 1646 external_mini_cluster.cc:1428] Setting key 938d8e3ee596173db5b800f117ac346b
I20260620 21:49:46.879225 23206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f: Bootstrap starting.
I20260620 21:49:46.879421 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:46.875077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:45935 (local address 127.0.0.1:41461)
0620 21:49:46.875514 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:46.875533 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:46.875578 (+ 45us) server_negotiation.cc:407] Connection header received
0620 21:49:46.876420 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:46.876443 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:46.876798 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:46.877165 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:46.878014 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:46.878020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:46.878036 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:46.878049 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:46.878115 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:46.878142 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:46.878295 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:46.878367 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 21:49:46.878520 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":117,"threads_started":1}
I20260620 21:49:46.881542 23206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:46.882290 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:46.882395 23206 log.cc:824] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:46.884701 23206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f: No bootstrap required, opened a new log
I20260620 21:49:46.886195 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:46.884457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:48385 (local address 127.0.0.1:41461)
0620 21:49:46.884585 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:46.884590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:46.884803 (+ 213us) server_negotiation.cc:407] Connection header received
0620 21:49:46.884891 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:46.884893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:46.884923 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:46.884989 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:46.885733 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:46.885736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:46.885738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:46.885749 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:46.885772 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:46.885782 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:46.885942 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:46.886024 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 21:49:46.886058 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:49:46.888732 23206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10293f3c7ca34856aa3226fde913213f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } }
I20260620 21:49:46.889009 23206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:46.889081 23206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10293f3c7ca34856aa3226fde913213f, State: Initialized, Role: FOLLOWER
I20260620 21:49:46.889559 23206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10293f3c7ca34856aa3226fde913213f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } }
I20260620 21:49:46.889745 23206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:46.889912 23206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:46.890002 23206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:46.890911 23206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10293f3c7ca34856aa3226fde913213f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } }
I20260620 21:49:46.891211 23206 leader_election.cc:302] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10293f3c7ca34856aa3226fde913213f; no voters:
I20260620 21:49:46.891567 23206 leader_election.cc:288] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:46.891898 23216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:46.892122 23216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [term 1 LEADER]: Becoming Leader. State: Replica: 10293f3c7ca34856aa3226fde913213f, State: Running, Role: LEADER
I20260620 21:49:46.892484 23216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10293f3c7ca34856aa3226fde913213f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } }
I20260620 21:49:46.892925 23206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:46.894413 23218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10293f3c7ca34856aa3226fde913213f. Latest consensus state: current_term: 1 leader_uuid: "10293f3c7ca34856aa3226fde913213f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10293f3c7ca34856aa3226fde913213f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } } }
I20260620 21:49:46.894431 23217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10293f3c7ca34856aa3226fde913213f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10293f3c7ca34856aa3226fde913213f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } } }
I20260620 21:49:46.894572 23217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:46.894572 23218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f [sys.catalog]: This master's current role is: LEADER
W20260620 21:49:46.898036 23231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:49:46.898136 23231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:49:46.898211 23228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:46.898969 23228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:46.904662 23228 catalog_manager.cc:1347] Generated new cluster ID: f6f70e4f55204c9191b806db90f68332
I20260620 21:49:46.904740 23228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:46.933943 23228 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:46.934322 23228 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:46.944332 23228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10293f3c7ca34856aa3226fde913213f: Generated new TSK 0
I20260620 21:49:46.945173 23228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:46.952772 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 23138
2026-06-20T21:49:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:47 dist-test-slave-5bvn krb5kdc[23237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:47 dist-test-slave-5bvn krb5kdc[23237](info): set up 2 sockets
Jun 20 21:49:47 dist-test-slave-5bvn krb5kdc[23237](info): commencing operation
krb5kdc: starting...
W20260620 21:49:47.998337 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:48 dist-test-slave-5bvn krb5kdc[23237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:49:48.124818 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:33663
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:36349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:33663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:48.237774 23250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:48.238078 23250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:48.238122 23250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:48.242286 23250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:48.242348 23250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:48.242367 23250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:48.242384 23250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:48.242424 23250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:48.247761 23250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:36349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:33663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:33663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:48.248909 23250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:48.249984 23250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:48.258556 23255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:48.258556 23258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:48.258559 23256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:48.258878 23250 server_base.cc:1034] running on GCE node
I20260620 21:49:48.259703 23250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:48.260823 23250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:48.261945 23250 hybrid_clock.cc:648] HybridClock initialized: now 1781992188261927 us; error 31 us; skew 500 ppm
Jun 20 21:49:48 dist-test-slave-5bvn krb5kdc[23237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992188, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:48.264583 23250 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:49:48.265398 23250 webserver.cc:460] Webserver started at http://127.1.155.190:40389/ using document root <none> and password file <none>
I20260620 21:49:48.265965 23250 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:48.266031 23250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:48.266196 23250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:48.268136 23250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "826aead71b594e2d883f479f80f7a775"
format_stamp: "Formatted at 2026-06-20 21:49:48 on dist-test-slave-5bvn"
server_key: "8da8235556e835f0dbf32a266b44fb13"
server_key_iv: "7e1af0cf718988d63ed8d683c34b5d21"
server_key_version: "encryptionkey@0"
I20260620 21:49:48.268646 23250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "826aead71b594e2d883f479f80f7a775"
format_stamp: "Formatted at 2026-06-20 21:49:48 on dist-test-slave-5bvn"
server_key: "8da8235556e835f0dbf32a266b44fb13"
server_key_iv: "7e1af0cf718988d63ed8d683c34b5d21"
server_key_version: "encryptionkey@0"
I20260620 21:49:48.272253 23250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:49:48.274669 23265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:48.275871 23250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:49:48.275964 23250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "826aead71b594e2d883f479f80f7a775"
format_stamp: "Formatted at 2026-06-20 21:49:48 on dist-test-slave-5bvn"
server_key: "8da8235556e835f0dbf32a266b44fb13"
server_key_iv: "7e1af0cf718988d63ed8d683c34b5d21"
server_key_version: "encryptionkey@0"
I20260620 21:49:48.276050 23250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:48.292255 23250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:48.295224 23250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:48.295388 23250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:48.302090 23250 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:33663
I20260620 21:49:48.302114 23317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:33663 every 8 connection(s)
I20260620 21:49:48.303031 23250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:48.305999 23318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:48.311308 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 23250
I20260620 21:49:48.311441 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:48.311605 23318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775: Bootstrap starting.
I20260620 21:49:48.311800 1646 external_mini_cluster.cc:1428] Setting key a782097f7cc21fdaf1d9000c416ed139
I20260620 21:49:48.314241 23318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:48.314985 23318 log.cc:824] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:48.316831 23318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775: No bootstrap required, opened a new log
Jun 20 21:49:48 dist-test-slave-5bvn krb5kdc[23237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:48.320672 23318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826aead71b594e2d883f479f80f7a775" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33663 } }
I20260620 21:49:48.320931 23318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:48.321012 23318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826aead71b594e2d883f479f80f7a775, State: Initialized, Role: FOLLOWER
I20260620 21:49:48.321473 23318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826aead71b594e2d883f479f80f7a775" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33663 } }
I20260620 21:49:48.321630 23318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:48.321739 23318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:48.321864 23318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:48.322914 23318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826aead71b594e2d883f479f80f7a775" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33663 } }
I20260620 21:49:48.323257 23318 leader_election.cc:302] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 826aead71b594e2d883f479f80f7a775; no voters:
I20260620 21:49:48.323597 23318 leader_election.cc:288] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:48.323663 23323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:48.323900 23323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [term 1 LEADER]: Becoming Leader. State: Replica: 826aead71b594e2d883f479f80f7a775, State: Running, Role: LEADER
I20260620 21:49:48.324251 23323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826aead71b594e2d883f479f80f7a775" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33663 } }
I20260620 21:49:48.324934 23318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:48.326067 23324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "826aead71b594e2d883f479f80f7a775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826aead71b594e2d883f479f80f7a775" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33663 } } }
I20260620 21:49:48.326184 23324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:48.326521 23325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 826aead71b594e2d883f479f80f7a775. Latest consensus state: current_term: 1 leader_uuid: "826aead71b594e2d883f479f80f7a775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826aead71b594e2d883f479f80f7a775" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33663 } } }
I20260620 21:49:48.326606 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:48.313205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34319 (local address 127.1.155.190:33663)
0620 21:49:48.313647 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:48.313662 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:48.313695 (+ 33us) server_negotiation.cc:407] Connection header received
0620 21:49:48.314471 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:48.314489 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:48.314841 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:48.315163 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:48.315949 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:48.316916 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:48.317790 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:48.318345 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:48.318513 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:48.321389 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:48.321393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:48.321407 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:48.321433 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:48.323218 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:48.323739 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:48.323745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:48.323752 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:48.323902 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:48.324302 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:48.324305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:48.324306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:48.324717 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:48.324949 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:48.325259 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 21:49:48.325513 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":140,"threads_started":1}
I20260620 21:49:48.326753 23325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:48.326623 23332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:48.330107 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:48.330566 23332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:48.336707 23332 catalog_manager.cc:1347] Generated new cluster ID: c4e8254eec344a7cac09b31c3c35193a
I20260620 21:49:48.336812 23332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:48.341370 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:48.332149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46627 (local address 127.1.155.190:33663)
0620 21:49:48.332259 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:48.332264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:48.332436 (+ 172us) server_negotiation.cc:407] Connection header received
0620 21:49:48.332587 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:48.332591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:48.332640 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:48.332728 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:48.333533 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:48.334381 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:48.335709 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:48.335968 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:48.336080 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:48.337822 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:48.337828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:48.337830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:48.337885 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:48.339564 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:48.340082 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:48.340086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:48.340088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:48.340173 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:48.340498 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:48.340501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:48.340502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:48.340708 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:48.340854 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:48.341100 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 21:49:48.341199 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 21:49:48.345039 23332 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:48.346156 23332 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:48.353463 23332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 826aead71b594e2d883f479f80f7a775: Generated new TSK 0
I20260620 21:49:48.354322 23332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:48.360484 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 23250
W20260620 21:49:48.365800 23235 connection.cc:535] client connection to 127.1.155.190:33663 recv error: Network error: failed to read from TLS socket (remote: 127.1.155.190:33663): Connection reset by peer (error 104)
2026-06-20T21:49:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:48 dist-test-slave-5bvn krb5kdc[23349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:48 dist-test-slave-5bvn krb5kdc[23349](info): set up 2 sockets
Jun 20 21:49:48 dist-test-slave-5bvn krb5kdc[23349](info): commencing operation
krb5kdc: starting...
W20260620 21:49:49.398176 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:49 dist-test-slave-5bvn krb5kdc[23349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:49:49.524930 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:37779
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:37779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:49.636011 23362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:49.636340 23362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:49.636415 23362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:49.640542 23362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:49.640648 23362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:49.640684 23362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:49.640741 23362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:49.640779 23362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:49.646186 23362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:37779
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:37779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:49.647289 23362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:49.648301 23362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:49.656890 23368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:49.656977 23362 server_base.cc:1034] running on GCE node
W20260620 21:49:49.656932 23367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:49.656921 23370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:49.657624 23362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:49.658736 23362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:49.659904 23362 hybrid_clock.cc:648] HybridClock initialized: now 1781992189659876 us; error 41 us; skew 500 ppm
Jun 20 21:49:49 dist-test-slave-5bvn krb5kdc[23349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992189, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:49.663075 23362 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:49:49.663954 23362 webserver.cc:460] Webserver started at http://127.1.155.190:35599/ using document root <none> and password file <none>
I20260620 21:49:49.664503 23362 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:49.664562 23362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:49.664774 23362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:49.666918 23362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "dee45337a9eb428fab2c2bae80ee9897"
format_stamp: "Formatted at 2026-06-20 21:49:49 on dist-test-slave-5bvn"
server_key: "a78108c689095f213456e258a08f7fd4"
server_key_iv: "08ec6fe8f17ad99d835f0c5048499869"
server_key_version: "encryptionkey@0"
I20260620 21:49:49.667481 23362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "dee45337a9eb428fab2c2bae80ee9897"
format_stamp: "Formatted at 2026-06-20 21:49:49 on dist-test-slave-5bvn"
server_key: "a78108c689095f213456e258a08f7fd4"
server_key_iv: "08ec6fe8f17ad99d835f0c5048499869"
server_key_version: "encryptionkey@0"
I20260620 21:49:49.671226 23362 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:49:49.673523 23377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:49.674757 23362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:49:49.674921 23362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "dee45337a9eb428fab2c2bae80ee9897"
format_stamp: "Formatted at 2026-06-20 21:49:49 on dist-test-slave-5bvn"
server_key: "a78108c689095f213456e258a08f7fd4"
server_key_iv: "08ec6fe8f17ad99d835f0c5048499869"
server_key_version: "encryptionkey@0"
I20260620 21:49:49.675038 23362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:49.729539 23362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:49.732757 23362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:49.732990 23362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:49.740626 23362 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:37779
I20260620 21:49:49.740639 23429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:37779 every 8 connection(s)
I20260620 21:49:49.741571 23362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:49.742003 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 23362
I20260620 21:49:49.742168 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:49.742451 1646 external_mini_cluster.cc:1428] Setting key 8dab22eca323750b1e7cc8728aa555fe
I20260620 21:49:49.744894 23430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:49.750716 23430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897: Bootstrap starting.
I20260620 21:49:49.751420 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:49.743885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55879 (local address 127.1.155.190:37779)
0620 21:49:49.744366 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:49.744392 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:49.744428 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:49:49.745096 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:49.745115 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:49.745469 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:49.745803 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:49.746233 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:49.747182 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:49.748057 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:49.748529 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:49.748714 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:49.749657 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:49.749663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:49.749675 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:49.750132 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:49.750190 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:49.750213 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:49.750353 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:49.750534 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 21:49:49.750674 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":150,"threads_started":1}
I20260620 21:49:49.753268 23430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:49.754132 23430 log.cc:824] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:49.754463 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:49.756402 23430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897: No bootstrap required, opened a new log
I20260620 21:49:49.761089 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:49.756528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45393 (local address 127.1.155.190:37779)
0620 21:49:49.756647 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:49.756651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:49.756856 (+ 205us) server_negotiation.cc:407] Connection header received
0620 21:49:49.756994 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:49.756996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:49.757033 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:49.757092 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:49.757495 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:49.758051 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:49.759399 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:49.759607 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:49.759688 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:49.760602 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:49.760605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:49.760607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:49.760620 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:49.760644 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:49.760654 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:49.760721 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:49.760880 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 21:49:49.760940 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:49:49.761669 23430 raft_consensus.cc:348] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee45337a9eb428fab2c2bae80ee9897" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 37779 } }
I20260620 21:49:49.761972 23430 raft_consensus.cc:374] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:49.762049 23430 raft_consensus.cc:729] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee45337a9eb428fab2c2bae80ee9897, State: Initialized, Role: FOLLOWER
I20260620 21:49:49.762473 23430 consensus_queue.cc:260] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee45337a9eb428fab2c2bae80ee9897" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 37779 } }
I20260620 21:49:49.762626 23430 raft_consensus.cc:388] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:49.762715 23430 raft_consensus.cc:482] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:49.762815 23430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:49.763799 23430 raft_consensus.cc:504] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee45337a9eb428fab2c2bae80ee9897" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 37779 } }
I20260620 21:49:49.764178 23430 leader_election.cc:302] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dee45337a9eb428fab2c2bae80ee9897; no voters:
I20260620 21:49:49.764518 23430 leader_election.cc:288] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:49.764701 23440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:49.765057 23440 raft_consensus.cc:686] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [term 1 LEADER]: Becoming Leader. State: Replica: dee45337a9eb428fab2c2bae80ee9897, State: Running, Role: LEADER
I20260620 21:49:49.765439 23440 consensus_queue.cc:237] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee45337a9eb428fab2c2bae80ee9897" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 37779 } }
I20260620 21:49:49.766003 23430 sys_catalog.cc:564] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:49.767314 23442 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dee45337a9eb428fab2c2bae80ee9897. Latest consensus state: current_term: 1 leader_uuid: "dee45337a9eb428fab2c2bae80ee9897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee45337a9eb428fab2c2bae80ee9897" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 37779 } } }
I20260620 21:49:49.767408 23441 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dee45337a9eb428fab2c2bae80ee9897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee45337a9eb428fab2c2bae80ee9897" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 37779 } } }
I20260620 21:49:49.767498 23441 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:49.767449 23442 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:49.767882 23451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:49.770943 23451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:49.776996 23451 catalog_manager.cc:1347] Generated new cluster ID: 1ce9211ea8d14798aeab9f9a1f7cebc9
I20260620 21:49:49.777098 23451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:49.813566 23451 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:49.814566 23451 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:49.820978 23451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dee45337a9eb428fab2c2bae80ee9897: Generated new TSK 0
I20260620 21:49:49.821723 23451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:49.828480 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 23362
W20260620 21:49:49.833928 23347 connection.cc:535] client connection to 127.1.155.190:37779 recv error: Network error: failed to read from TLS socket (remote: 127.1.155.190:37779): Connection reset by peer (error 104)
2026-06-20T21:49:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:49 dist-test-slave-5bvn krb5kdc[23461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:49 dist-test-slave-5bvn krb5kdc[23461](info): set up 2 sockets
Jun 20 21:49:49 dist-test-slave-5bvn krb5kdc[23461](info): commencing operation
krb5kdc: starting...
W20260620 21:49:50.882154 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:50 dist-test-slave-5bvn krb5kdc[23461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:49:51.009303 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:44205
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:44205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:51.119953 23474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:51.120319 23474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:51.120397 23474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:51.124588 23474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:51.124709 23474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:51.124750 23474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:51.124789 23474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:51.124825 23474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:51.130488 23474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:44205
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:44205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:51.131691 23474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:51.132730 23474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:51.141294 23479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:51.141278 23480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:51.141403 23482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:51.141539 23474 server_base.cc:1034] running on GCE node
I20260620 21:49:51.142220 23474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:51.143348 23474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:51.144552 23474 hybrid_clock.cc:648] HybridClock initialized: now 1781992191144535 us; error 37 us; skew 500 ppm
Jun 20 21:49:51 dist-test-slave-5bvn krb5kdc[23461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992191, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:51.147344 23474 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:49:51.148244 23474 webserver.cc:460] Webserver started at http://127.1.155.190:41521/ using document root <none> and password file <none>
I20260620 21:49:51.148803 23474 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:51.148905 23474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:51.149134 23474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:51.151259 23474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "5d1ff30a322b4969bff282f365e097b9"
format_stamp: "Formatted at 2026-06-20 21:49:51 on dist-test-slave-5bvn"
server_key: "11441c329d65e7b4a81fecf8e697572f"
server_key_iv: "5e11ce9969220027e48738f3a75b55b2"
server_key_version: "encryptionkey@0"
I20260620 21:49:51.151841 23474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "5d1ff30a322b4969bff282f365e097b9"
format_stamp: "Formatted at 2026-06-20 21:49:51 on dist-test-slave-5bvn"
server_key: "11441c329d65e7b4a81fecf8e697572f"
server_key_iv: "5e11ce9969220027e48738f3a75b55b2"
server_key_version: "encryptionkey@0"
I20260620 21:49:51.155661 23474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:49:51.158121 23489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:51.159384 23474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:49:51.159536 23474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "5d1ff30a322b4969bff282f365e097b9"
format_stamp: "Formatted at 2026-06-20 21:49:51 on dist-test-slave-5bvn"
server_key: "11441c329d65e7b4a81fecf8e697572f"
server_key_iv: "5e11ce9969220027e48738f3a75b55b2"
server_key_version: "encryptionkey@0"
I20260620 21:49:51.159657 23474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:51.164247 23474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:51.167233 23474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:51.167425 23474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:51.174037 23474 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:44205
I20260620 21:49:51.174091 23541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:44205 every 8 connection(s)
I20260620 21:49:51.175000 23474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:51.175966 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 23474
I20260620 21:49:51.176105 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:51.176409 1646 external_mini_cluster.cc:1428] Setting key 3b6e3618b74fcd9e8235c6d2ccbd7d05
I20260620 21:49:51.178541 23542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:51.182031 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:51.177745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54877 (local address 127.1.155.190:44205)
0620 21:49:51.178226 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:51.178246 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:51.178291 (+ 45us) server_negotiation.cc:407] Connection header received
0620 21:49:51.179142 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:51.179165 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:51.179525 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:51.179881 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:51.180634 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:51.180640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:51.180655 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:51.180667 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:51.180729 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:51.180753 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:51.180886 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:51.180951 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 21:49:51.181097 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":130,"threads_started":1}
I20260620 21:49:51.185036 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:51.185341 23542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9: Bootstrap starting.
I20260620 21:49:51.187825 23542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:51.188738 23542 log.cc:824] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:51.188788 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:51.187189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40475 (local address 127.1.155.190:44205)
0620 21:49:51.187300 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:51.187304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:51.187455 (+ 151us) server_negotiation.cc:407] Connection header received
0620 21:49:51.187550 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:51.187553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:51.187588 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:51.187676 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:51.188427 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:51.188431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:51.188434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:51.188446 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:51.188474 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:51.188486 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:51.188578 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:51.188638 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 21:49:51.188674 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 21:49:51.190841 23542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9: No bootstrap required, opened a new log
I20260620 21:49:51.194509 23542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1ff30a322b4969bff282f365e097b9" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 44205 } }
I20260620 21:49:51.194738 23542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:51.194780 23542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1ff30a322b4969bff282f365e097b9, State: Initialized, Role: FOLLOWER
I20260620 21:49:51.195204 23542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1ff30a322b4969bff282f365e097b9" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 44205 } }
I20260620 21:49:51.195327 23542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:51.195374 23542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:51.195444 23542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:51.196336 23542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1ff30a322b4969bff282f365e097b9" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 44205 } }
I20260620 21:49:51.196691 23542 leader_election.cc:302] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d1ff30a322b4969bff282f365e097b9; no voters:
I20260620 21:49:51.197017 23542 leader_election.cc:288] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:51.197182 23552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:51.197458 23552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [term 1 LEADER]: Becoming Leader. State: Replica: 5d1ff30a322b4969bff282f365e097b9, State: Running, Role: LEADER
I20260620 21:49:51.197791 23552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1ff30a322b4969bff282f365e097b9" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 44205 } }
I20260620 21:49:51.198441 23542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:51.199661 23553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d1ff30a322b4969bff282f365e097b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1ff30a322b4969bff282f365e097b9" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 44205 } } }
I20260620 21:49:51.199668 23554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d1ff30a322b4969bff282f365e097b9. Latest consensus state: current_term: 1 leader_uuid: "5d1ff30a322b4969bff282f365e097b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1ff30a322b4969bff282f365e097b9" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 44205 } } }
I20260620 21:49:51.199811 23553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:51.199851 23554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:51.200287 23562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:51.203320 23562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:51.209163 23562 catalog_manager.cc:1347] Generated new cluster ID: 813f0cbee89c43feb75aee151908ecef
I20260620 21:49:51.209267 23562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:51.217288 23562 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:51.217638 23562 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:51.224267 23562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d1ff30a322b4969bff282f365e097b9: Generated new TSK 0
I20260620 21:49:51.224963 23562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:51.231745 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 23474
2026-06-20T21:49:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:51 dist-test-slave-5bvn krb5kdc[23573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:51 dist-test-slave-5bvn krb5kdc[23573](info): set up 2 sockets
Jun 20 21:49:51 dist-test-slave-5bvn krb5kdc[23573](info): commencing operation
krb5kdc: starting...
W20260620 21:49:52.334182 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:52 dist-test-slave-5bvn krb5kdc[23573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:49:52.456935 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:38017
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:38017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:52.572003 23586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:52.572324 23586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:52.572372 23586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:52.576609 23586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:52.576699 23586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:52.576723 23586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:52.576753 23586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:52.576819 23586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:52.582509 23586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:38017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:38017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:52.583647 23586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:52.584718 23586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:49:52.593508 23586 server_base.cc:1034] running on GCE node
W20260620 21:49:52.593470 23594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:52.593488 23591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:52.593470 23592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:52.594283 23586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:52.595352 23586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:52.596534 23586 hybrid_clock.cc:648] HybridClock initialized: now 1781992192596514 us; error 36 us; skew 500 ppm
Jun 20 21:49:52 dist-test-slave-5bvn krb5kdc[23573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992192, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:52.602830 23586 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:49:52.603886 23586 webserver.cc:460] Webserver started at http://127.1.155.190:41505/ using document root <none> and password file <none>
I20260620 21:49:52.604465 23586 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:52.604559 23586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:52.604795 23586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:52.606971 23586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "6c8b0b77551048dca5829816a70f5616"
format_stamp: "Formatted at 2026-06-20 21:49:52 on dist-test-slave-5bvn"
server_key: "7c5488b040d12fe50aaf8de25e25edf7"
server_key_iv: "84f2635dc43fdd4a21222306de9f06ca"
server_key_version: "encryptionkey@0"
I20260620 21:49:52.607555 23586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "6c8b0b77551048dca5829816a70f5616"
format_stamp: "Formatted at 2026-06-20 21:49:52 on dist-test-slave-5bvn"
server_key: "7c5488b040d12fe50aaf8de25e25edf7"
server_key_iv: "84f2635dc43fdd4a21222306de9f06ca"
server_key_version: "encryptionkey@0"
I20260620 21:49:52.611357 23586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:49:52.614252 23601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:52.615800 23586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:49:52.615990 23586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "6c8b0b77551048dca5829816a70f5616"
format_stamp: "Formatted at 2026-06-20 21:49:52 on dist-test-slave-5bvn"
server_key: "7c5488b040d12fe50aaf8de25e25edf7"
server_key_iv: "84f2635dc43fdd4a21222306de9f06ca"
server_key_version: "encryptionkey@0"
I20260620 21:49:52.616127 23586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:52.636489 23586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:52.639827 23586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:52.640051 23586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:52.647405 23586 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:38017
I20260620 21:49:52.647493 23653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:38017 every 8 connection(s)
I20260620 21:49:52.648492 23586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:52.651511 23654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:52.654367 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 23586
I20260620 21:49:52.654534 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:52.654827 1646 external_mini_cluster.cc:1428] Setting key 567ea29a6afb05cf2085a7c8740fc7dd
I20260620 21:49:52.658351 23654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616: Bootstrap starting.
I20260620 21:49:52.660802 23654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:52.661549 23654 log.cc:824] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:52.663465 23654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616: No bootstrap required, opened a new log
Jun 20 21:49:52 dist-test-slave-5bvn krb5kdc[23573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:52.666690 23654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8b0b77551048dca5829816a70f5616" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38017 } }
I20260620 21:49:52.666944 23654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:52.667023 23654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c8b0b77551048dca5829816a70f5616, State: Initialized, Role: FOLLOWER
I20260620 21:49:52.667500 23654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8b0b77551048dca5829816a70f5616" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38017 } }
I20260620 21:49:52.667652 23654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:52.667734 23654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:52.667831 23654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:52.668779 23654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8b0b77551048dca5829816a70f5616" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38017 } }
I20260620 21:49:52.669126 23654 leader_election.cc:302] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c8b0b77551048dca5829816a70f5616; no voters:
I20260620 21:49:52.669531 23654 leader_election.cc:288] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:52.669623 23659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:52.669875 23659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [term 1 LEADER]: Becoming Leader. State: Replica: 6c8b0b77551048dca5829816a70f5616, State: Running, Role: LEADER
I20260620 21:49:52.670229 23659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8b0b77551048dca5829816a70f5616" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38017 } }
I20260620 21:49:52.670600 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:52.656521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43081 (local address 127.1.155.190:38017)
0620 21:49:52.657031 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:52.657046 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:52.657082 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:49:52.657707 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:52.657724 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:52.658178 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:52.658538 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:52.659377 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:52.660358 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:52.661262 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:52.661789 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:52.662021 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:52.664780 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:52.664786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:52.664800 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:52.664827 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:52.667359 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:52.667843 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:52.667851 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:52.667862 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:52.668026 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:52.668314 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:52.668318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:52.668320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:52.668725 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:52.668943 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:52.669228 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 21:49:52.669563 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":141,"threads_started":1}
I20260620 21:49:52.670898 23654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:52.673743 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:52.673525 23661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c8b0b77551048dca5829816a70f5616. Latest consensus state: current_term: 1 leader_uuid: "6c8b0b77551048dca5829816a70f5616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8b0b77551048dca5829816a70f5616" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38017 } } }
I20260620 21:49:52.673803 23660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c8b0b77551048dca5829816a70f5616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8b0b77551048dca5829816a70f5616" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 38017 } } }
I20260620 21:49:52.674041 23661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:52.674067 23660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616 [sys.catalog]: This master's current role is: LEADER
W20260620 21:49:52.678134 23673 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:49:52.678222 23673 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:49:52.678304 23675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:52.679153 23675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:52.684432 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:52.675924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53259 (local address 127.1.155.190:38017)
0620 21:49:52.676046 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:52.676050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:52.676156 (+ 106us) server_negotiation.cc:407] Connection header received
0620 21:49:52.676276 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:52.676278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:52.676316 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:52.676381 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:52.677195 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:52.678034 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:52.679168 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:52.679541 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:52.679655 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:52.681290 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:52.681294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:52.681296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:52.681320 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:52.682756 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:52.683168 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:52.683173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:52.683175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:52.683252 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:52.683541 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:52.683544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:52.683546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:52.683724 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:52.683857 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:52.684200 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 21:49:52.684320 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:49:52.686311 23675 catalog_manager.cc:1347] Generated new cluster ID: 0099375fb9844b409c25c10a61426e3f
I20260620 21:49:52.686369 23675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:52.725011 23675 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:52.726066 23675 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:52.732466 23675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c8b0b77551048dca5829816a70f5616: Generated new TSK 0
I20260620 21:49:52.733155 23675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:52.737089 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 23586
2026-06-20T21:49:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1505 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:52 dist-test-slave-5bvn krb5kdc[23685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:52 dist-test-slave-5bvn krb5kdc[23685](info): set up 2 sockets
Jun 20 21:49:52 dist-test-slave-5bvn krb5kdc[23685](info): commencing operation
krb5kdc: starting...
W20260620 21:49:53.794230 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:53 dist-test-slave-5bvn krb5kdc[23685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:49:53.920499 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:43235
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:43235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:54.035442 23698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:54.035760 23698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:54.035804 23698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:54.040208 23698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:54.040306 23698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:54.040328 23698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:54.040346 23698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:54.040364 23698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:54.046211 23698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:43235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:43235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:54.047587 23698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:54.048656 23698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:54.057694 23703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:54.057679 23704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:54.057698 23706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:54.058029 23698 server_base.cc:1034] running on GCE node
I20260620 21:49:54.058640 23698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:54.059965 23698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:54.061203 23698 hybrid_clock.cc:648] HybridClock initialized: now 1781992194061174 us; error 46 us; skew 500 ppm
Jun 20 21:49:54 dist-test-slave-5bvn krb5kdc[23685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992194, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:54.064131 23698 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:49:54.065115 23698 webserver.cc:460] Webserver started at http://127.1.155.190:42809/ using document root <none> and password file <none>
I20260620 21:49:54.065742 23698 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:54.065838 23698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:54.066102 23698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:54.068193 23698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "f1816dad365f49009aa8e87136b67620"
format_stamp: "Formatted at 2026-06-20 21:49:54 on dist-test-slave-5bvn"
server_key: "484dd7ce25160f6af7075e83c628033b"
server_key_iv: "444205848a3e7c5a661dbfdb6c2992e2"
server_key_version: "encryptionkey@0"
I20260620 21:49:54.068773 23698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "f1816dad365f49009aa8e87136b67620"
format_stamp: "Formatted at 2026-06-20 21:49:54 on dist-test-slave-5bvn"
server_key: "484dd7ce25160f6af7075e83c628033b"
server_key_iv: "444205848a3e7c5a661dbfdb6c2992e2"
server_key_version: "encryptionkey@0"
I20260620 21:49:54.072906 23698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 21:49:54.075912 23713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:54.077546 23698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:49:54.077750 23698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "f1816dad365f49009aa8e87136b67620"
format_stamp: "Formatted at 2026-06-20 21:49:54 on dist-test-slave-5bvn"
server_key: "484dd7ce25160f6af7075e83c628033b"
server_key_iv: "444205848a3e7c5a661dbfdb6c2992e2"
server_key_version: "encryptionkey@0"
I20260620 21:49:54.077919 23698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:54.100163 23698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:54.103552 23698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:54.103792 23698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:54.111191 23698 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:43235
I20260620 21:49:54.111213 23765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:43235 every 8 connection(s)
I20260620 21:49:54.112151 23698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:54.114882 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:54.119616 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 23698
I20260620 21:49:54.119767 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:54.120134 1646 external_mini_cluster.cc:1428] Setting key 6267fde40f3c2540dd2d74a9ec022911
I20260620 21:49:54.120538 23766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620: Bootstrap starting.
I20260620 21:49:54.122983 23766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:54.123688 23766 log.cc:824] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:54.125557 23766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620: No bootstrap required, opened a new log
I20260620 21:49:54.130110 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:54.121527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44655 (local address 127.1.155.190:43235)
0620 21:49:54.122056 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:54.122072 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:54.122106 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:49:54.122719 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:54.122734 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:54.123071 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:54.123444 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:54.123884 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:54.124958 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:54.125962 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:54.126440 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:54.126618 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:54.127918 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:54.127923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:54.127957 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:54.128353 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:54.128407 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:54.128430 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:54.128545 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:54.128702 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 21:49:54.128918 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":139,"threads_started":1}
I20260620 21:49:54.130420 23766 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1816dad365f49009aa8e87136b67620" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43235 } }
I20260620 21:49:54.130725 23766 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:54.130782 23766 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1816dad365f49009aa8e87136b67620, State: Initialized, Role: FOLLOWER
I20260620 21:49:54.131258 23766 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1816dad365f49009aa8e87136b67620" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43235 } }
I20260620 21:49:54.131440 23766 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:54.131507 23766 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:54.131601 23766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:54.132819 23766 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1816dad365f49009aa8e87136b67620" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43235 } }
I20260620 21:49:54.133164 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:54.133239 23766 leader_election.cc:302] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1816dad365f49009aa8e87136b67620; no voters:
I20260620 21:49:54.133590 23766 leader_election.cc:288] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:54.133749 23771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:54.133994 23771 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [term 1 LEADER]: Becoming Leader. State: Replica: f1816dad365f49009aa8e87136b67620, State: Running, Role: LEADER
I20260620 21:49:54.134399 23771 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1816dad365f49009aa8e87136b67620" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43235 } }
I20260620 21:49:54.134929 23766 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:54.135906 23773 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1816dad365f49009aa8e87136b67620. Latest consensus state: current_term: 1 leader_uuid: "f1816dad365f49009aa8e87136b67620" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1816dad365f49009aa8e87136b67620" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43235 } } }
I20260620 21:49:54.136075 23773 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:54.136318 23772 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1816dad365f49009aa8e87136b67620" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1816dad365f49009aa8e87136b67620" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 43235 } } }
I20260620 21:49:54.136415 23772 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:54.137639 23785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:54.141034 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:54.135719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55849 (local address 127.1.155.190:43235)
0620 21:49:54.135825 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:54.135829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:54.135844 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:49:54.136485 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:54.136488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:54.136545 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:54.136605 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:54.137061 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:54.137891 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:54.139049 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:54.139363 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:54.139453 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:54.140511 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:54.140514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:54.140517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:54.140531 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:54.140561 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:54.140574 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:54.140624 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:54.140814 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 21:49:54.140872 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 21:49:54.141156 23785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:54.148072 23785 catalog_manager.cc:1347] Generated new cluster ID: ff3baba4eba34169add79615df2515a0
I20260620 21:49:54.148152 23785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:54.164650 23785 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:54.166030 23785 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:54.176733 23785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1816dad365f49009aa8e87136b67620: Generated new TSK 0
I20260620 21:49:54.177623 23785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:54.185696 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 23698
2026-06-20T21:49:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1448 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:54 dist-test-slave-5bvn krb5kdc[23797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:54 dist-test-slave-5bvn krb5kdc[23797](info): set up 2 sockets
Jun 20 21:49:54 dist-test-slave-5bvn krb5kdc[23797](info): commencing operation
krb5kdc: starting...
W20260620 21:49:55.230289 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:55 dist-test-slave-5bvn krb5kdc[23797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:49:55.356606 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:33495
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:38573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:33495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:55.467247 23810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:55.467547 23810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:55.467593 23810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:55.471547 23810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:55.471613 23810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:55.471637 23810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:55.471657 23810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:55.471674 23810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:55.476909 23810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:38573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:33495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:33495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:55.477944 23810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:55.479130 23810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:55.487798 23815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:55.487846 23810 server_base.cc:1034] running on GCE node
W20260620 21:49:55.487769 23818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:55.487782 23816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:55.488603 23810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:55.489639 23810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:55.490837 23810 hybrid_clock.cc:648] HybridClock initialized: now 1781992195490834 us; error 63 us; skew 500 ppm
Jun 20 21:49:55 dist-test-slave-5bvn krb5kdc[23797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:55.493657 23810 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:49:55.494640 23810 webserver.cc:460] Webserver started at http://127.1.155.190:34895/ using document root <none> and password file <none>
I20260620 21:49:55.495199 23810 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:55.495292 23810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:55.495514 23810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:55.498081 23810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "6cb3c823826744d1be8f726a68bb94f2"
format_stamp: "Formatted at 2026-06-20 21:49:55 on dist-test-slave-5bvn"
server_key: "6e7998e870e52b8b1e9a078c9f974145"
server_key_iv: "a50c7ae7877522aa98c8a8950cb09392"
server_key_version: "encryptionkey@0"
I20260620 21:49:55.498670 23810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "6cb3c823826744d1be8f726a68bb94f2"
format_stamp: "Formatted at 2026-06-20 21:49:55 on dist-test-slave-5bvn"
server_key: "6e7998e870e52b8b1e9a078c9f974145"
server_key_iv: "a50c7ae7877522aa98c8a8950cb09392"
server_key_version: "encryptionkey@0"
I20260620 21:49:55.502429 23810 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:49:55.504879 23825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:55.506253 23810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:49:55.506397 23810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "6cb3c823826744d1be8f726a68bb94f2"
format_stamp: "Formatted at 2026-06-20 21:49:55 on dist-test-slave-5bvn"
server_key: "6e7998e870e52b8b1e9a078c9f974145"
server_key_iv: "a50c7ae7877522aa98c8a8950cb09392"
server_key_version: "encryptionkey@0"
I20260620 21:49:55.506512 23810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:55.511819 23810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:55.515340 23810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:55.515563 23810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:55.522562 23810 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:33495
I20260620 21:49:55.522589 23877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:33495 every 8 connection(s)
I20260620 21:49:55.523552 23810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:55.526329 23878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:55.531749 23878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2: Bootstrap starting.
I20260620 21:49:55.533329 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 23810
I20260620 21:49:55.533502 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:55.533774 1646 external_mini_cluster.cc:1428] Setting key 4453b2c25acf01a134b02da6b5bd6b6f
I20260620 21:49:55.534267 23878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:55.535172 23878 log.cc:824] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:55.537204 23878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2: No bootstrap required, opened a new log
I20260620 21:49:55.538949 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:55.534894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47051 (local address 127.1.155.190:33495)
0620 21:49:55.535453 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:55.535468 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:55.535503 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:49:55.536328 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:55.536347 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:55.536629 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:55.536977 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:55.537729 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:55.537735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:55.537749 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:55.537766 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:55.537814 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:55.537878 (+ 64us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:55.537981 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:55.538100 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 21:49:55.538258 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":125,"threads_started":1}
I20260620 21:49:55.541483 23878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb3c823826744d1be8f726a68bb94f2" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33495 } }
I20260620 21:49:55.541769 23878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:49:55.541824 23878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb3c823826744d1be8f726a68bb94f2, State: Initialized, Role: FOLLOWER
I20260620 21:49:55.542217 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:55.542352 23878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb3c823826744d1be8f726a68bb94f2" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33495 } }
I20260620 21:49:55.542503 23878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:49:55.542560 23878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:49:55.542649 23878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:55.543967 23878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb3c823826744d1be8f726a68bb94f2" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33495 } }
I20260620 21:49:55.544309 23878 leader_election.cc:302] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cb3c823826744d1be8f726a68bb94f2; no voters:
I20260620 21:49:55.544684 23878 leader_election.cc:288] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:49:55.544775 23888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:55.545006 23888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [term 1 LEADER]: Becoming Leader. State: Replica: 6cb3c823826744d1be8f726a68bb94f2, State: Running, Role: LEADER
I20260620 21:49:55.545395 23888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb3c823826744d1be8f726a68bb94f2" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33495 } }
I20260620 21:49:55.546137 23878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:55.546651 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:55.544461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55411 (local address 127.1.155.190:33495)
0620 21:49:55.544594 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:55.544598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:55.544798 (+ 200us) server_negotiation.cc:407] Connection header received
0620 21:49:55.545004 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:55.545007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:55.545053 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:55.545139 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:55.546052 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:55.546055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:55.546058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:49:55.546071 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:55.546098 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:49:55.546111 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:55.546170 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:55.546471 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 21:49:55.546510 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:49:55.546783 23888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb3c823826744d1be8f726a68bb94f2. Latest consensus state: current_term: 1 leader_uuid: "6cb3c823826744d1be8f726a68bb94f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb3c823826744d1be8f726a68bb94f2" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33495 } } }
I20260620 21:49:55.546911 23888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:55.547192 23890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cb3c823826744d1be8f726a68bb94f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb3c823826744d1be8f726a68bb94f2" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33495 } } }
I20260620 21:49:55.547310 23890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:55.547338 23893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:55.550603 23893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:55.556423 23893 catalog_manager.cc:1347] Generated new cluster ID: 6d880b2dd1ab429dae864c7860f88209
I20260620 21:49:55.556527 23893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:55.584240 23893 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:55.584729 23893 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:55.593053 23893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cb3c823826744d1be8f726a68bb94f2: Generated new TSK 0
I20260620 21:49:55.593933 23893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:49:55.609362 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 23810
2026-06-20T21:49:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1424 ms)
[----------] 18 tests from AuthTokenIssuingTest (25991 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:55 dist-test-slave-5bvn krb5kdc[23909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:55 dist-test-slave-5bvn krb5kdc[23909](info): set up 2 sockets
Jun 20 21:49:55 dist-test-slave-5bvn krb5kdc[23909](info): commencing operation
krb5kdc: starting...
W20260620 21:49:56.666234 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:56 dist-test-slave-5bvn krb5kdc[23909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:49:56.790884 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:33393
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:38303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:33393,127.1.155.189:44213,127.1.155.188:46431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:49:56.904835 23922 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:49:56.905189 23922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:56.905241 23922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:56.905313 23922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:49:56.910153 23922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:56.910286 23922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:56.910323 23922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:56.910367 23922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:56.910405 23922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:56.916255 23922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:38303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:33393,127.1.155.189:44213,127.1.155.188:46431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:33393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:56.917433 23922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:56.918470 23922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:56.927491 23928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:56.927533 23930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:56.927512 23922 server_base.cc:1034] running on GCE node
W20260620 21:49:56.927507 23927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:56.928421 23922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:56.929574 23922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:56.930780 23922 hybrid_clock.cc:648] HybridClock initialized: now 1781992196930740 us; error 54 us; skew 500 ppm
Jun 20 21:49:56 dist-test-slave-5bvn krb5kdc[23909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:56.933660 23922 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:49:56.934780 23922 webserver.cc:460] Webserver started at http://127.1.155.190:44781/ using document root <none> and password file <none>
I20260620 21:49:56.935370 23922 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:56.935465 23922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:56.935678 23922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:56.937892 23922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "98e4779ef86a449794ad4f7b770ac021"
format_stamp: "Formatted at 2026-06-20 21:49:56 on dist-test-slave-5bvn"
server_key: "7ea27cec7ebf9d5c638fa41e98249bdb"
server_key_iv: "f2b192a208702ba392d2c47ff674f4c0"
server_key_version: "encryptionkey@0"
I20260620 21:49:56.938503 23922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "98e4779ef86a449794ad4f7b770ac021"
format_stamp: "Formatted at 2026-06-20 21:49:56 on dist-test-slave-5bvn"
server_key: "7ea27cec7ebf9d5c638fa41e98249bdb"
server_key_iv: "f2b192a208702ba392d2c47ff674f4c0"
server_key_version: "encryptionkey@0"
I20260620 21:49:56.942524 23922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:49:56.945050 23937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:56.946447 23922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 21:49:56.946605 23922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "98e4779ef86a449794ad4f7b770ac021"
format_stamp: "Formatted at 2026-06-20 21:49:56 on dist-test-slave-5bvn"
server_key: "7ea27cec7ebf9d5c638fa41e98249bdb"
server_key_iv: "f2b192a208702ba392d2c47ff674f4c0"
server_key_version: "encryptionkey@0"
I20260620 21:49:56.946733 23922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:56.961159 23922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:56.964562 23922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:56.964725 23922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:56.972635 23922 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:33393
I20260620 21:49:56.972939 23989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:33393 every 8 connection(s)
I20260620 21:49:56.973630 23922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:49:56.976423 23990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:56.977563 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 23922
I20260620 21:49:56.977710 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:49:56.978082 1646 external_mini_cluster.cc:1428] Setting key 548856c65495b77649a58e34b20eb1f1
I20260620 21:49:56.980224 23990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.155.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.189@KRBTEST.COM" created.
Jun 20 21:49:56 dist-test-slave-5bvn krb5kdc[23909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:56.995357 23993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:56.982511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.1.155.190:33393)
0620 21:49:56.982919 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:56.982933 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:56.983261 (+ 328us) server_negotiation.cc:407] Connection header received
0620 21:49:56.984275 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:56.984280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:56.984458 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:56.984646 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:56.985490 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:56.986522 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:56.987492 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:56.987858 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:56.988027 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:56.990434 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:56.990444 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:56.990461 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:56.990498 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:56.992140 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:56.992739 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:56.992743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:56.992745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:56.992846 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:56.993233 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:56.993235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:56.993237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:56.993617 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:56.993719 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:56.994239 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 21:49:56.994515 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":258,"thread_start_us":121,"threads_started":1}
I20260620 21:49:56.995360 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:56.982512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:33393 (local address 127.0.0.1:57506)
0620 21:49:56.983121 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:56.983155 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:56.983974 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:56.984642 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:56.984646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:56.985077 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:56.985306 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:56.985344 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:56.986648 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:56.986652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:56.987368 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:56.987377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:56.988012 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:56.988016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:56.988124 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:56.988731 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:56.988748 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:56.990272 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:56.992257 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:56.992272 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:56.992285 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:56.992626 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:56.992947 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:56.992953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:56.992956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:56.993129 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:56.993749 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:56.993759 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:56.993950 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 21:49:56.994158 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 21:49:56.994608 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":83,"thread_start_us":157,"threads_started":1}
I20260620 21:49:56.997763 23990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:49:56.998592 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:56.998335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:44213 (local address 127.0.0.1:40814)
0620 21:49:56.998452 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:56.998523 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:44213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 21:49:56.998854 23938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.155.189:44213: connect: Connection refused (error 111)
I20260620 21:49:56.999858 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:56.999636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:44213 (local address 127.0.0.1:40826)
0620 21:49:56.999756 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:56.999804 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:44213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 21:49:57.000102 23990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.189:44213: Network error: Client connection negotiation failed: client connection to 127.1.155.189:44213: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.189.keytab.
Entry for principal kudu/127.1.155.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.189.keytab.
I20260620 21:49:57.004626 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.189:44213
--webserver_interface=127.1.155.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.189.keytab
--principal=kudu/127.1.155.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:38303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:33393,127.1.155.189:44213,127.1.155.188:46431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:49:57.049384 23990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } attempt: 1
I20260620 21:49:57.050395 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.050058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:44213 (local address 127.0.0.1:40842)
0620 21:49:57.050245 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.050308 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:44213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 21:49:57.051219 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.050991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:44213 (local address 127.0.0.1:40854)
0620 21:49:57.051108 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.051161 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:44213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 21:49:57.051338 23990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.189:44213: Network error: Client connection negotiation failed: client connection to 127.1.155.189:44213: connect: Connection refused (error 111)
W20260620 21:49:57.119102 23997 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:49:57.119463 23997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:57.119529 23997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:57.119607 23997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 21:49:57.119518 23990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } attempt: 2
I20260620 21:49:57.120355 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.120059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:44213 (local address 127.0.0.1:40858)
0620 21:49:57.120215 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.120282 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:44213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 21:49:57.121110 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.120894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:44213 (local address 127.0.0.1:40862)
0620 21:49:57.121010 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.121060 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:44213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 21:49:57.121237 23990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.189:44213: Network error: Client connection negotiation failed: client connection to 127.1.155.189:44213: connect: Connection refused (error 111)
W20260620 21:49:57.124092 23997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:57.124177 23997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:57.124223 23997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:57.124258 23997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:57.124315 23997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:57.129889 23997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:38303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:33393,127.1.155.189:44213,127.1.155.188:46431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.189.keytab
--principal=kudu/127.1.155.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.189:44213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.155.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:57.131076 23997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:57.132092 23997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:49:57.140688 24005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:57.140866 23997 server_base.cc:1034] running on GCE node
W20260620 21:49:57.140713 24002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:57.140770 24003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:57.141665 23997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:57.142858 23997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:57.144089 23997 hybrid_clock.cc:648] HybridClock initialized: now 1781992197144058 us; error 49 us; skew 500 ppm
Jun 20 21:49:57 dist-test-slave-5bvn krb5kdc[23909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:57.146890 23997 init.cc:377] Logged in from keytab as kudu/127.1.155.189@KRBTEST.COM (short username kudu)
I20260620 21:49:57.147876 23997 webserver.cc:460] Webserver started at http://127.1.155.189:34267/ using document root <none> and password file <none>
I20260620 21:49:57.148442 23997 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:57.148538 23997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:57.148767 23997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:57.151014 23997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/data/instance:
uuid: "91a46323a9c44fcc95347d916da06f57"
format_stamp: "Formatted at 2026-06-20 21:49:57 on dist-test-slave-5bvn"
server_key: "5e645a443830850a73222203e38d577a"
server_key_iv: "b5038383312481a22db9f0f5fc3cd3d0"
server_key_version: "encryptionkey@0"
I20260620 21:49:57.151571 23997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/wal/instance:
uuid: "91a46323a9c44fcc95347d916da06f57"
format_stamp: "Formatted at 2026-06-20 21:49:57 on dist-test-slave-5bvn"
server_key: "5e645a443830850a73222203e38d577a"
server_key_iv: "b5038383312481a22db9f0f5fc3cd3d0"
server_key_version: "encryptionkey@0"
I20260620 21:49:57.155370 23997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:49:57.158042 24012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:57.159394 23997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:49:57.159560 23997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/wal
uuid: "91a46323a9c44fcc95347d916da06f57"
format_stamp: "Formatted at 2026-06-20 21:49:57 on dist-test-slave-5bvn"
server_key: "5e645a443830850a73222203e38d577a"
server_key_iv: "b5038383312481a22db9f0f5fc3cd3d0"
server_key_version: "encryptionkey@0"
I20260620 21:49:57.159699 23997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:57.174160 23997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:57.177103 23997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:57.177326 23997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:57.185000 23997 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.189:44213
I20260620 21:49:57.185256 24064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.189:44213 every 8 connection(s)
I20260620 21:49:57.186074 23997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/data/info.pb
I20260620 21:49:57.188794 24065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:57.191421 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 23997
I20260620 21:49:57.191586 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-1/wal/instance
I20260620 21:49:57.191884 1646 external_mini_cluster.cc:1428] Setting key 744e706e121aaf2059080829c9a77d50
I20260620 21:49:57.192757 24065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.155.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.188@KRBTEST.COM" created.
Jun 20 21:49:57 dist-test-slave-5bvn krb5kdc[23909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.189@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:57.208920 23993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.194676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.1.155.190:33393)
0620 21:49:57.194840 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.194844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.195700 (+ 856us) server_negotiation.cc:407] Connection header received
0620 21:49:57.196626 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.196632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.196701 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.196822 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.198020 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.198916 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.200291 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.200548 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.200641 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.203452 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:57.203455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:57.203460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:57.203492 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:57.206054 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.206709 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.206716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.206717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.206817 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.207380 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.207386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.207389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.207614 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:57.207917 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:57.208554 (+ 637us) server_negotiation.cc:299] Negotiation successful
0620 21:49:57.208704 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:49:57.209815 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.195061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:33393 (local address 127.0.0.1:57518)
0620 21:49:57.195529 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.195581 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:57.196408 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:57.196963 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:57.196969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:57.197299 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:57.197798 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.197811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.199092 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.199095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.200088 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.200099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.200721 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.200725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.200891 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.201640 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:57.201657 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:57.203269 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:57.206205 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.206211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.206226 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.206546 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.207004 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.207011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.207013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.207179 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.207909 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:57.207917 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:57.208245 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 21:49:57.208472 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 21:49:57.208844 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":142,"threads_started":1}
I20260620 21:49:57.211489 24065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:49:57.212461 23990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } attempt: 3
Jun 20 21:49:57 dist-test-slave-5bvn krb5kdc[23909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.189@KRBTEST.COM for kudu/127.1.155.189@KRBTEST.COM
Jun 20 21:49:57 dist-test-slave-5bvn krb5kdc[23909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for kudu/127.1.155.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.188.keytab.
Entry for principal kudu/127.1.155.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.188.keytab.
I20260620 21:49:57.222110 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.188:46431
--webserver_interface=127.1.155.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.188.keytab
--principal=kudu/127.1.155.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:38303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:33393,127.1.155.189:44213,127.1.155.188:46431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:49:57.224279 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.212861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:44213 (local address 127.0.0.1:40878)
0620 21:49:57.213030 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.213044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:57.213114 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:57.213580 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:57.213582 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:57.213723 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:57.213822 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.213827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.215213 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.215217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.216147 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.216158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.217342 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.217346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.217460 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.218194 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:57.218213 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:57.220130 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:57.222532 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.222536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.222538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.222751 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.223113 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.223119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.223124 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.223230 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.223717 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:57.223720 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:57.223830 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 21:49:57.223939 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 21:49:57.224097 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 21:49:57.224360 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.212963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.1.155.189:44213)
0620 21:49:57.213338 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.213351 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.213362 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:49:57.213390 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.213392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.213426 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.213574 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.214031 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.215059 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.216279 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.217191 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.217332 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.220294 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:57.220297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:57.220299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:57.220321 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:57.222376 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.222868 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.222874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.222876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.222958 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.223308 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.223312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.223314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.223599 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:57.223736 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:57.224042 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 21:49:57.224185 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":567,"server-negotiator.queue_time_us":308,"spinlock_wait_cycles":16256,"thread_start_us":103,"threads_started":1}
I20260620 21:49:57.225073 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.212068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:44213 (local address 127.0.0.1:40872)
0620 21:49:57.212209 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.212222 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:57.212314 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:57.213133 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:57.213139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:57.213333 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:57.213482 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.213491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.215257 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.215263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.216138 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.216151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.217342 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.217346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.217460 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.218254 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:57.218273 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:57.219753 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:57.222681 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.222687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.222692 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.222931 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.223233 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.223237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.223239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.223320 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.224570 (+ 1250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:57.224573 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:57.224645 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 21:49:57.224755 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 21:49:57.224893 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"spinlock_wait_cycles":12672}
I20260620 21:49:57.225097 24070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.212240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.1.155.189:44213)
0620 21:49:57.212677 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.212689 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.212713 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:49:57.212761 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.212770 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.212978 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.213574 (+ 596us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.213796 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.215059 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.216279 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.217191 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.217350 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.219903 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:57.219913 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:57.219932 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:57.219960 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:57.222563 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.223046 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.223052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.223055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.223133 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.223401 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.223406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.223408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.223652 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:57.223743 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:57.224808 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0620 21:49:57.224949 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":191,"server-negotiator.queue_time_us":309,"spinlock_wait_cycles":13568,"thread_start_us":187,"threads_started":1}
I20260620 21:49:57.226711 24065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:49:57.227177 23990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:49:57.227421 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.227166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:46431 (local address 127.0.0.1:42070)
0620 21:49:57.227278 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.227354 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 21:49:57.227602 24013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
I20260620 21:49:57.227998 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.227633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:46431 (local address 127.0.0.1:42082)
0620 21:49:57.227834 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.227893 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 21:49:57.228623 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.228377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:46431 (local address 127.0.0.1:42084)
0620 21:49:57.228513 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.228559 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 21:49:57.228726 24065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:46431: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
I20260620 21:49:57.228804 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.228485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:46431 (local address 127.0.0.1:42100)
0620 21:49:57.228659 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.228720 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260620 21:49:57.228942 23990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:46431: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
I20260620 21:49:57.260141 23990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } attempt: 1
I20260620 21:49:57.261059 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.260717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:46431 (local address 127.0.0.1:42112)
0620 21:49:57.260901 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.260972 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 21:49:57.261914 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.261658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:46431 (local address 127.0.0.1:42114)
0620 21:49:57.261759 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.261804 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 21:49:57.262116 23990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:46431: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
I20260620 21:49:57.278025 24065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } attempt: 1
I20260620 21:49:57.278980 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.278663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:46431 (local address 127.0.0.1:42130)
0620 21:49:57.278822 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.278898 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 21:49:57.279743 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.279523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:46431 (local address 127.0.0.1:42146)
0620 21:49:57.279633 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.279688 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 21:49:57.279870 24065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:46431: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
W20260620 21:49:57.336524 24072 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:49:57.336869 24072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:49:57.336939 24072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:49:57.337041 24072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 21:49:57.337313 23990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } attempt: 2
I20260620 21:49:57.338166 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.337893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:46431 (local address 127.0.0.1:42154)
0620 21:49:57.338031 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.338091 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 21:49:57.338907 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.338702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:46431 (local address 127.0.0.1:42166)
0620 21:49:57.338806 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.338856 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 21:49:57.339062 23990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:46431: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
W20260620 21:49:57.341637 24072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:49:57.341689 24072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:49:57.341709 24072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:49:57.341727 24072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:49:57.341784 24072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:49:57.347167 24072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:38303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:33393,127.1.155.189:44213,127.1.155.188:46431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.188.keytab
--principal=kudu/127.1.155.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.188:46431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.155.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:49:57.348033 24065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } attempt: 2
I20260620 21:49:57.348232 24072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:49:57.348824 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.348530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:46431 (local address 127.0.0.1:42168)
0620 21:49:57.348681 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.348746 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 21:49:57.349504 24072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:49:57.349563 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.349320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:46431 (local address 127.0.0.1:42172)
0620 21:49:57.349428 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.349479 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 21:49:57.349717 24065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:46431: Network error: Client connection negotiation failed: client connection to 127.1.155.188:46431: connect: Connection refused (error 111)
W20260620 21:49:57.358068 24081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:49:57.358068 24078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:57.358192 24072 server_base.cc:1034] running on GCE node
W20260620 21:49:57.358124 24079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:49:57.358939 24072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:49:57.359987 24072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:49:57.361155 24072 hybrid_clock.cc:648] HybridClock initialized: now 1781992197361124 us; error 45 us; skew 500 ppm
Jun 20 21:49:57 dist-test-slave-5bvn krb5kdc[23909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:49:57.363973 24072 init.cc:377] Logged in from keytab as kudu/127.1.155.188@KRBTEST.COM (short username kudu)
I20260620 21:49:57.364878 24072 webserver.cc:460] Webserver started at http://127.1.155.188:41221/ using document root <none> and password file <none>
I20260620 21:49:57.365444 24072 fs_manager.cc:359] Metadata directory not provided
I20260620 21:49:57.365551 24072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:49:57.365773 24072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:49:57.367769 24072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/data/instance:
uuid: "cdc081a70a8040488dd6cb342f92600d"
format_stamp: "Formatted at 2026-06-20 21:49:57 on dist-test-slave-5bvn"
server_key: "51b6e25b64bc412d92572505a005ee2d"
server_key_iv: "f5427d01167b1587f2a72693ef531b91"
server_key_version: "encryptionkey@0"
I20260620 21:49:57.368326 24072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/wal/instance:
uuid: "cdc081a70a8040488dd6cb342f92600d"
format_stamp: "Formatted at 2026-06-20 21:49:57 on dist-test-slave-5bvn"
server_key: "51b6e25b64bc412d92572505a005ee2d"
server_key_iv: "f5427d01167b1587f2a72693ef531b91"
server_key_version: "encryptionkey@0"
I20260620 21:49:57.372056 24072 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:49:57.374537 24088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:49:57.375921 24072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:49:57.376091 24072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/wal
uuid: "cdc081a70a8040488dd6cb342f92600d"
format_stamp: "Formatted at 2026-06-20 21:49:57 on dist-test-slave-5bvn"
server_key: "51b6e25b64bc412d92572505a005ee2d"
server_key_iv: "f5427d01167b1587f2a72693ef531b91"
server_key_version: "encryptionkey@0"
I20260620 21:49:57.376214 24072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:49:57.427263 24072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:49:57.430572 24072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:49:57.430776 24072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:49:57.437597 24072 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.188:46431
I20260620 21:49:57.437956 24140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.188:46431 every 8 connection(s)
I20260620 21:49:57.438213 23990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } attempt: 3
I20260620 21:49:57.438690 24072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/data/info.pb
I20260620 21:49:57.440712 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 24072
I20260620 21:49:57.440841 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781991920067886-1646-0/minicluster-data/master-2/wal/instance
I20260620 21:49:57.440908 24065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } attempt: 3
I20260620 21:49:57.441138 1646 external_mini_cluster.cc:1428] Setting key 7b9cc8714e966b07b87d0f2f8a2fc407
I20260620 21:49:57.441785 24141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:49:57.447082 24141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:49:57 dist-test-slave-5bvn krb5kdc[23909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for kudu/127.1.155.188@KRBTEST.COM
Jun 20 21:49:57 dist-test-slave-5bvn krb5kdc[23909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.189@KRBTEST.COM for kudu/127.1.155.188@KRBTEST.COM
Jun 20 21:49:57 dist-test-slave-5bvn krb5kdc[23909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.188@KRBTEST.COM
I20260620 21:49:57.454325 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.438732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:46431 (local address 127.0.0.1:42188)
0620 21:49:57.438939 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.438957 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:57.439040 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:57.441520 (+ 2480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:57.441523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:57.441743 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:57.441921 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.441929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.444359 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.444362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.445154 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.445165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.446290 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.446294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.446394 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.447049 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:57.447062 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:57.448688 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:57.452140 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.452144 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.452146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.452660 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.452995 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.452998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.453000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.453091 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.453807 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:57.453812 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:57.453931 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 21:49:57.454007 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 21:49:57.454133 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 21:49:57.454380 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.441310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:46431 (local address 127.0.0.1:42202)
0620 21:49:57.441531 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.441548 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:57.441628 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:57.442284 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:57.442288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:57.442467 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:57.442585 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.442592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.444364 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.444367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.445186 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.445198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.446301 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.446305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.446388 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.447298 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:57.447316 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:57.449562 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:57.452273 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.452279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.452282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.452559 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.452991 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.452994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.452997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.453117 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.453795 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:57.453801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:57.453939 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 21:49:57.454034 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 21:49:57.454200 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260620 21:49:57.456091 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.442343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42218 (local address 127.1.155.188:46431)
0620 21:49:57.442910 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.442919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.442930 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:49:57.442983 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.442985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.443022 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.443102 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.444055 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.444824 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.446003 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.446337 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.446477 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.450209 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:57.450266 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:57.450271 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:57.450312 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:57.453215 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.453704 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.453710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.453714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.453824 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.454169 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.454173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.454175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.454353 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:57.454549 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:57.455241 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 21:49:57.455340 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":505,"thread_start_us":76,"threads_started":1}
I20260620 21:49:57.456099 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.441540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42202 (local address 127.1.155.188:46431)
0620 21:49:57.442030 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.442042 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.442058 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:49:57.442096 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.442098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.442141 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.442696 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.442801 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.444220 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.445483 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.446166 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.447956 (+ 1790us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.449749 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:57.449753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:57.449755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:57.449779 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:57.452124 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.452713 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.452721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.452729 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.452868 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.453228 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.453232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.453235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.453679 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:57.454642 (+ 963us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:57.454810 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 21:49:57.455171 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":416,"spinlock_wait_cycles":8576,"thread_start_us":112,"threads_started":1}
I20260620 21:49:57.456091 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.439477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.1.155.188:46431)
0620 21:49:57.439865 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.439891 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.439924 (+ 33us) server_negotiation.cc:407] Connection header received
0620 21:49:57.440649 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.440676 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.441141 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.442149 (+ 1008us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.442609 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.444217 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.445292 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.446159 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.446633 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.448904 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:57.448912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:57.448932 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:57.448982 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:57.452021 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.452773 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.452776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.452778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.452884 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.453205 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.453208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.453209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.453674 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:57.454641 (+ 967us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:57.454807 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 21:49:57.455265 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":508,"server-negotiator.queue_time_us":255,"spinlock_wait_cycles":26880,"thread_start_us":142,"threads_started":1}
Jun 20 21:49:57 dist-test-slave-5bvn krb5kdc[23909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.188@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:57.460358 23990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021: Bootstrap starting.
I20260620 21:49:57.462023 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.449003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:33393 (local address 127.0.0.1:57534)
0620 21:49:57.450353 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.450404 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:57.450596 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:57.450905 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:57.450911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:57.451518 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:57.451690 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.451700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.452773 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.452776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.453725 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.453736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.454779 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.454785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.454889 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.456203 (+ 1314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:57.456405 (+ 202us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:57.458132 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:57.460052 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.460061 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.460076 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.460350 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.460702 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.460705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.460707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.460797 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.461338 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:57.461343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:57.461468 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 21:49:57.461631 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 21:49:57.461818 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"thread_start_us":85,"threads_started":1}
I20260620 21:49:57.462023 23993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.449030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.1.155.190:33393)
0620 21:49:57.449162 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.449166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.450478 (+ 1312us) server_negotiation.cc:407] Connection header received
0620 21:49:57.450702 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.450707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.450762 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.450853 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.451837 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.452635 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.453861 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.454174 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.454243 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.458257 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:57.458261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:57.458263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:57.458290 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:57.459937 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.460495 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.460500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.460504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.460593 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.460906 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.460909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.460911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.461220 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:57.461317 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:57.461702 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 21:49:57.461806 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:49:57.463126 23990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:57.463126 24141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:49:57.464030 23990 log.cc:824] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:49:57 dist-test-slave-5bvn krb5kdc[23909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:49:57.466935 24065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57: Bootstrap starting.
Jun 20 21:49:57 dist-test-slave-5bvn krb5kdc[23909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.188@KRBTEST.COM for kudu/127.1.155.189@KRBTEST.COM
I20260620 21:49:57.469714 24065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:57.470118 24148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.459138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.1.155.190:33393)
0620 21:49:57.459533 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.459546 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.459562 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:49:57.460159 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.460162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.460206 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.460302 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.461139 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.461831 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.462968 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.463315 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.463416 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.466395 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:57.466399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:57.466402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:57.466425 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:57.468117 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.468566 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.468570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.468572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.468659 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.468962 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.468966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.468967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.469155 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:57.469368 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:57.469743 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 21:49:57.469904 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":100,"threads_started":1}
I20260620 21:49:57.470623 24065 log.cc:824] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:57.472501 24065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57: No bootstrap required, opened a new log
I20260620 21:49:57.472502 23990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021: No bootstrap required, opened a new log
I20260620 21:49:57.472993 24070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.463729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.1.155.189:44213)
0620 21:49:57.463870 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.463875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.463931 (+ 56us) server_negotiation.cc:407] Connection header received
0620 21:49:57.464053 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.464059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.464112 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.464194 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.464780 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.465472 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.466617 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.467000 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.467135 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.469271 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:57.469280 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:57.469283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:57.469309 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:57.471006 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.471485 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.471488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.471491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.471578 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.471903 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.471907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.471909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.472194 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:57.472260 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:57.472697 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 21:49:57.472838 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:49:57.473531 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.463625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:44213 (local address 127.0.0.1:40886)
0620 21:49:57.463864 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.463881 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:57.463966 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:57.464326 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:57.464328 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:57.464512 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:57.464634 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.464644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.465653 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.465657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.466480 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.466488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.467277 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.467281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.467395 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.467815 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:57.467830 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:57.469132 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:57.471120 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.471124 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.471126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.471368 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.471701 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.471704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.471707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.471796 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.472332 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:57.472335 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:57.472438 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 21:49:57.473234 (+ 796us) client_negotiation.cc:240] Negotiation successful
0620 21:49:57.473378 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260620 21:49:57.474320 24141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:49:57.476660 23990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } }
I20260620 21:49:57.476897 23990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98e4779ef86a449794ad4f7b770ac021, State: Initialized, Role: FOLLOWER
Jun 20 21:49:57 dist-test-slave-5bvn krb5kdc[23909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.189@KRBTEST.COM
I20260620 21:49:57.477236 23990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } }
I20260620 21:49:57.477522 24065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } }
I20260620 21:49:57.477728 24065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a46323a9c44fcc95347d916da06f57, State: Initialized, Role: FOLLOWER
I20260620 21:49:57.477933 24151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } } }
I20260620 21:49:57.478048 24065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } }
I20260620 21:49:57.478223 24151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:49:57.478735 23990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:57.479266 24065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:57.482002 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.471760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.1.155.189:44213)
0620 21:49:57.471889 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.471893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.472041 (+ 148us) server_negotiation.cc:407] Connection header received
0620 21:49:57.472200 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.472203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.472249 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.472298 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.473172 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.473979 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.474975 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.475326 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.475383 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.477933 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:57.477937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:57.477941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:57.477968 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:57.479682 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.480187 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.480191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.480192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.480285 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.480581 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.480585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.480587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.480815 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:57.481012 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:57.481695 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 21:49:57.481811 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 21:49:57.485055 24174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:49:57.485152 24174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 20 21:49:57 dist-test-slave-5bvn krb5kdc[23909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.188@KRBTEST.COM for kudu/127.1.155.188@KRBTEST.COM
I20260620 21:49:57.490337 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.474796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:46431 (local address 127.0.0.1:42232)
0620 21:49:57.475032 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 21:49:57.475045 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:49:57.475116 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:49:57.481349 (+ 6233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:49:57.481355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:49:57.481560 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:49:57.481682 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.481691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.482796 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.482799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.483523 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:49:57.483534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.484086 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.484089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:49:57.484163 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.484633 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:49:57.484647 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:49:57.486634 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:49:57.488685 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.488689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.488691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.488938 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.489225 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:49:57.489228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:49:57.489229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:49:57.489309 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:49:57.489757 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:49:57.489762 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:49:57.489913 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 21:49:57.490020 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 21:49:57.490169 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260620 21:49:57.490761 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.475031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.1.155.188:46431)
0620 21:49:57.481108 (+ 6077us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.481113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.481128 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:49:57.481166 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.481170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.481215 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.481800 (+ 585us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.481911 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.482677 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.483660 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.483972 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.484372 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.486779 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:57.486782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:57.486785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:57.486810 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:57.488561 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.489045 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.489049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.489051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.489128 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.489408 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.489411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.489413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.489630 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:57.489729 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:57.490520 (+ 791us) server_negotiation.cc:299] Negotiation successful
0620 21:49:57.490633 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5993}
I20260620 21:49:57.488976 23922 master_runner.cc:186] Error getting master registration for 127.1.155.188:46431: OK, instance_id { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" instance_seqno: 1781992197435804 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:49:57.492853 23922 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 21:49:57.493114 24168 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:49:57.493181 24168 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:49:57.493384 24152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } } }
I20260620 21:49:57.493506 24152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:49:57.496577 24141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d: Bootstrap starting.
I20260620 21:49:57.497340 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:49:57.498871 23963 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "98e4779ef86a449794ad4f7b770ac021"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57548
I20260620 21:49:57.499107 23963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 21:49:57.499217 23963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:57.499450 24141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d: Neither blocks nor log segments found. Creating new log.
I20260620 21:49:57.500319 24141 log.cc:824] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:49:57.501065 23963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } }
I20260620 21:49:57.502620 24039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98e4779ef86a449794ad4f7b770ac021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "91a46323a9c44fcc95347d916da06f57"
I20260620 21:49:57.502871 24039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:57.503660 24141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d: No bootstrap required, opened a new log
I20260620 21:49:57.504175 24039 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98e4779ef86a449794ad4f7b770ac021 in term 1.
I20260620 21:49:57.504822 23938 leader_election.cc:302] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91a46323a9c44fcc95347d916da06f57, 98e4779ef86a449794ad4f7b770ac021; no voters:
I20260620 21:49:57.505151 24151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:49:57.505385 24151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [term 1 LEADER]: Becoming Leader. State: Replica: 98e4779ef86a449794ad4f7b770ac021, State: Running, Role: LEADER
I20260620 21:49:57.505738 24151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } }
I20260620 21:49:57.507565 24151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98e4779ef86a449794ad4f7b770ac021. Latest consensus state: current_term: 1 leader_uuid: "98e4779ef86a449794ad4f7b770ac021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } } }
I20260620 21:49:57.507705 24151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:57.508199 23963 leader_election.cc:288] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [CANDIDATE]: Term 1 election: Requested vote from peers 91a46323a9c44fcc95347d916da06f57 (127.1.155.189:44213), cdc081a70a8040488dd6cb342f92600d (127.1.155.188:46431)
I20260620 21:49:57.508244 24141 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } }
I20260620 21:49:57.508519 24141 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc081a70a8040488dd6cb342f92600d, State: Initialized, Role: FOLLOWER
I20260620 21:49:57.508848 24141 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } }
I20260620 21:49:57.510218 24141 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:49:57.510833 24114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98e4779ef86a449794ad4f7b770ac021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cdc081a70a8040488dd6cb342f92600d"
I20260620 21:49:57.511051 24114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:49:57.512440 24114 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98e4779ef86a449794ad4f7b770ac021 in term 1.
I20260620 21:49:57.512795 24180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:49:57.513788 24180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:49:57.493986 23997 master_runner.cc:186] Error getting master registration for 127.1.155.188:46431: OK, instance_id { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" instance_seqno: 1781992197435804 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:49:57.514226 23997 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:49:57.516173 24181 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } } }
I20260620 21:49:57.516306 24181 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:49:57.521999 24114 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d [term 1 FOLLOWER]: Refusing update from remote peer 98e4779ef86a449794ad4f7b770ac021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:49:57.522294 24039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57 [term 1 FOLLOWER]: Refusing update from remote peer 98e4779ef86a449794ad4f7b770ac021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:49:57.522802 24178 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:49:57.523734 24151 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:49:57.528468 24152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98e4779ef86a449794ad4f7b770ac021. Latest consensus state: current_term: 1 leader_uuid: "98e4779ef86a449794ad4f7b770ac021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } } }
I20260620 21:49:57.528617 24152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:49:57.528765 24181 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98e4779ef86a449794ad4f7b770ac021. Latest consensus state: current_term: 1 leader_uuid: "98e4779ef86a449794ad4f7b770ac021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } } }
I20260620 21:49:57.528880 24181 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:49:57.535997 24148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.516816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.1.155.190:33393)
0620 21:49:57.516935 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.516940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.526179 (+ 9239us) server_negotiation.cc:407] Connection header received
0620 21:49:57.526375 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.526379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.526430 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.526487 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.527561 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.528395 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.529819 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.530249 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.530326 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.532281 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:57.532285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:57.532288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:57.532322 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:57.534064 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.534538 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.534541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.534543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.534626 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.534915 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.534918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.534920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.535143 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:57.535254 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:57.535695 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 21:49:57.535828 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 21:49:57.536998 24151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98e4779ef86a449794ad4f7b770ac021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } } }
I20260620 21:49:57.537161 24151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:57.537828 24197 mvcc.cc:205] Tried to move back new op lower bound from 7299040041043599360 to 7299040040987185152. Current Snapshot: MvccSnapshot[applied={T|T < 7299040041043599360}]
I20260620 21:49:57.538401 24152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98e4779ef86a449794ad4f7b770ac021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } } }
I20260620 21:49:57.538537 24152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:49:57.538949 24203 mvcc.cc:205] Tried to move back new op lower bound from 7299040041043599360 to 7299040040987185152. Current Snapshot: MvccSnapshot[applied={T|T < 7299040041043599360}]
I20260620 21:49:57.540467 24180 catalog_manager.cc:1347] Generated new cluster ID: acccb777f57448449816f9b7ce610516
I20260620 21:49:57.540537 24180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:49:57.545805 24178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98e4779ef86a449794ad4f7b770ac021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } } }
I20260620 21:49:57.546002 24178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021 [sys.catalog]: This master's current role is: LEADER
I20260620 21:49:57.547380 24181 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98e4779ef86a449794ad4f7b770ac021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e4779ef86a449794ad4f7b770ac021" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 33393 } } peers { permanent_uuid: "91a46323a9c44fcc95347d916da06f57" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 44213 } } peers { permanent_uuid: "cdc081a70a8040488dd6cb342f92600d" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 46431 } } }
I20260620 21:49:57.547511 24181 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:49:57.547745 24201 mvcc.cc:205] Tried to move back new op lower bound from 7299040041043599360 to 7299040040987185152. Current Snapshot: MvccSnapshot[applied={T|T < 7299040041043599360}]
I20260620 21:49:57.555408 24196 catalog_manager.cc:1259] Loaded cluster ID: acccb777f57448449816f9b7ce610516
I20260620 21:49:57.555476 24196 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d: loading cluster ID for follower catalog manager: success
W20260620 21:49:57.555994 24196 catalog_manager.cc:1576] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260620 21:49:57.556083 24196 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:49:57.560274 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.517330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.1.155.189:44213)
0620 21:49:57.517441 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.517446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.529253 (+ 11807us) server_negotiation.cc:407] Connection header received
0620 21:49:57.529422 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.529425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.529475 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.529528 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.531936 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.532713 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.535611 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.535936 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.536015 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.549890 (+ 13875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:57.549895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:57.549898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:57.549934 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:57.551693 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.556245 (+ 4552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.556250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.556251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.556356 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.559021 (+ 2665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.559025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.559027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.559268 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:57.559382 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:57.559971 (+ 589us) server_negotiation.cc:299] Negotiation successful
0620 21:49:57.560106 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:49:57.564013 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:57.517920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42244 (local address 127.1.155.188:46431)
0620 21:49:57.518024 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.518028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.530699 (+ 12671us) server_negotiation.cc:407] Connection header received
0620 21:49:57.530914 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.530918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.530982 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.531032 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.533878 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.534548 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.536676 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:57.537009 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:57.537093 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:57.556515 (+ 19422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:49:57.556519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:49:57.556522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:49:57.556552 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:49:57.558297 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.559704 (+ 1407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.559708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.559710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.559810 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:49:57.562929 (+ 3119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:49:57.562932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:49:57.562934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:49:57.563177 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:49:57.563283 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:57.563704 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 21:49:57.563829 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 21:49:57.583014 24180 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:49:57.584354 24180 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:49:57.605975 24180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98e4779ef86a449794ad4f7b770ac021: Generated new TSK 0
I20260620 21:49:57.606961 24180 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:49:57.640594 24148 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:49:57.637366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.1.155.190:33393)
0620 21:49:57.637530 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.637535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.639388 (+ 1853us) server_negotiation.cc:407] Connection header received
0620 21:49:57.639631 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.639635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.639694 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.639751 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.640441 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57558: BlockingRecv error: recv got EOF from 127.0.0.1:57558 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 21:49:57.642035 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:49:57.638060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.1.155.189:44213)
0620 21:49:57.638185 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.638190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.641125 (+ 2935us) server_negotiation.cc:407] Connection header received
0620 21:49:57.641295 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.641299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.641348 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.641398 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.641921 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40910: BlockingRecv error: recv got EOF from 127.0.0.1:40910 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 21:49:57.643478 24146 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:49:57.638761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42256 (local address 127.1.155.188:46431)
0620 21:49:57.638878 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.638883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.642509 (+ 3626us) server_negotiation.cc:407] Connection header received
0620 21:49:57.642712 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.642715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.642769 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.642824 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.643355 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42256: BlockingRecv error: recv got EOF from 127.0.0.1:42256 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 21:49:57.648473 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:49:57.646725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.1.155.189:44213)
0620 21:49:57.646846 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.646851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.647439 (+ 588us) server_negotiation.cc:407] Connection header received
0620 21:49:57.647686 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.647689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.647769 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.647825 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.648361 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40914: BlockingRecv error: recv got EOF from 127.0.0.1:40914 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 21:49:57.656427 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:49:57.654671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.1.155.189:44213)
0620 21:49:57.654793 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.654798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.655334 (+ 536us) server_negotiation.cc:407] Connection header received
0620 21:49:57.655557 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.655560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.655627 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.655686 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.656313 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40920: BlockingRecv error: recv got EOF from 127.0.0.1:40920 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 21:49:57.662688 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:49:57.660884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40932 (local address 127.1.155.189:44213)
0620 21:49:57.660998 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.661002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.661492 (+ 490us) server_negotiation.cc:407] Connection header received
0620 21:49:57.661707 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.661711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.661766 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.661821 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.662534 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40932: BlockingRecv error: recv got EOF from 127.0.0.1:40932 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 21:49:57.670964 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:49:57.669210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.1.155.189:44213)
0620 21:49:57.669365 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.669370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.669940 (+ 570us) server_negotiation.cc:407] Connection header received
0620 21:49:57.670154 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.670158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.670215 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.670271 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.670849 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40944: BlockingRecv error: recv got EOF from 127.0.0.1:40944 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 21:49:57.687237 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:49:57.682260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.1.155.189:44213)
0620 21:49:57.682378 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.682383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.686215 (+ 3832us) server_negotiation.cc:407] Connection header received
0620 21:49:57.686424 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.686427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.686483 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.686537 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.687111 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40960: BlockingRecv error: recv got EOF from 127.0.0.1:40960 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 21:49:57.707402 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:49:57.706078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.1.155.189:44213)
0620 21:49:57.706242 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.706246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.706406 (+ 160us) server_negotiation.cc:407] Connection header received
0620 21:49:57.706614 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.706617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.706665 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.706730 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.707275 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40970: BlockingRecv error: recv got EOF from 127.0.0.1:40970 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 21:49:57.742766 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:49:57.741606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40976 (local address 127.1.155.189:44213)
0620 21:49:57.741748 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.741753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.741943 (+ 190us) server_negotiation.cc:407] Connection header received
0620 21:49:57.742074 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.742078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.742132 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.742230 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.742643 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40976: BlockingRecv error: recv got EOF from 127.0.0.1:40976 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 21:49:57.810508 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:49:57.809316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40980 (local address 127.1.155.189:44213)
0620 21:49:57.809492 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.809496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.809583 (+ 87us) server_negotiation.cc:407] Connection header received
0620 21:49:57.809701 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.809703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.809753 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.809950 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.810404 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40980: BlockingRecv error: recv got EOF from 127.0.0.1:40980 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 21:49:57.941830 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:49:57.940621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40994 (local address 127.1.155.189:44213)
0620 21:49:57.940781 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:57.940786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:57.940923 (+ 137us) server_negotiation.cc:407] Connection header received
0620 21:49:57.941133 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:57.941136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:57.941183 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:57.941275 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:57.941711 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40994: BlockingRecv error: recv got EOF from 127.0.0.1:40994 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 21:49:58.201229 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:49:58.200026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.1.155.189:44213)
0620 21:49:58.200188 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:58.200193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:58.200303 (+ 110us) server_negotiation.cc:407] Connection header received
0620 21:49:58.200492 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:58.200495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:58.200541 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:58.200636 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:49:58.201094 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41004: BlockingRecv error: recv got EOF from 127.0.0.1:41004 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 21:49:58.486475 24174 catalog_manager.cc:1259] Loaded cluster ID: acccb777f57448449816f9b7ce610516
I20260620 21:49:58.486564 24174 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57: loading cluster ID for follower catalog manager: success
I20260620 21:49:58.488260 24174 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57: acquiring CA information for follower catalog manager: success
I20260620 21:49:58.489428 24174 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91a46323a9c44fcc95347d916da06f57: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:49:58.558658 24196 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d: acquiring CA information for follower catalog manager: success
I20260620 21:49:58.559916 24196 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cdc081a70a8040488dd6cb342f92600d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:49:58.719255 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:58.715395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.1.155.189:44213)
0620 21:49:58.715610 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:58.715614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:58.715632 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:49:58.715773 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:58.715776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:58.715902 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:58.716007 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:49:58.716257 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:58.716779 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:58.717628 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:58.717884 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:58.717979 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:58.718428 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:49:58.718868 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:49:58.718960 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:58.719080 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 21:49:58.719132 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
I20260620 21:49:58.726034 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:58.722398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.1.155.188:46431)
0620 21:49:58.722647 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:58.722651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:58.722664 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:49:58.722741 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:58.722743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:58.722855 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:58.722960 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:49:58.723143 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:58.723705 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:58.724518 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:58.724759 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:58.724854 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:58.725142 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:49:58.725586 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:49:58.725699 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:58.725762 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 21:49:58.725814 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":78,"threads_started":1}
I20260620 21:49:58.733654 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:58.729494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.1.155.189:44213)
0620 21:49:58.729655 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:58.729661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:58.729680 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:49:58.729912 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:58.729919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:58.729979 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:58.730025 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:49:58.730375 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:58.731252 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:58.732398 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:58.732668 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:58.732762 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:58.733071 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:49:58.733163 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:49:58.733243 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:58.733425 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 21:49:58.733483 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:49:58.733932 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:58.729952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42276 (local address 127.1.155.188:46431)
0620 21:49:58.730088 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:58.730096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:58.730139 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:49:58.730306 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:58.730311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:58.730374 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:58.730456 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:49:58.730700 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:58.731599 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:58.732656 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:58.732975 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:58.733074 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:58.733341 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:49:58.733415 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:49:58.733494 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:58.733699 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 21:49:58.733757 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:49:58.734465 24285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:49:58.729191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.1.155.190:33393)
0620 21:49:58.729451 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 21:49:58.729458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:49:58.729475 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:49:58.729576 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:49:58.729579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:49:58.729766 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:49:58.730178 (+ 412us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:49:58.730291 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:58.730984 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:58.732242 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:49:58.732635 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:49:58.732739 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:49:58.733077 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:49:58.733454 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:49:58.734183 (+ 729us) server_negotiation.cc:1035] Waiting for connection context
0620 21:49:58.734226 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 21:49:58.734298 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260620 21:49:58.738046 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 23922
I20260620 21:49:58.744721 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 23997
I20260620 21:49:58.750779 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 24072
2026-06-20T21:49:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3142 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:49:58 dist-test-slave-5bvn krb5kdc[24290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:49:58 dist-test-slave-5bvn krb5kdc[24290](info): set up 2 sockets
Jun 20 21:49:58 dist-test-slave-5bvn krb5kdc[24290](info): commencing operation
krb5kdc: starting...
W20260620 21:49:59.794269 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/test-user.keytab.
Jun 20 21:49:59 dist-test-slave-5bvn krb5kdc[24290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab.
I20260620 21:49:59.916993 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:36259
--webserver_interface=127.1.155.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:36259,127.1.155.189:35009,127.1.155.188:38835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:00.039749 24303 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:50:00.040300 24303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:00.040373 24303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:00.040472 24303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:00.046761 24303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:00.047174 24303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:00.047297 24303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:00.047341 24303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:00.047379 24303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:00.055011 24303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:36259,127.1.155.189:35009,127.1.155.188:38835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.190.keytab
--principal=kudu/127.1.155.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.190:36259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.155.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:00.056564 24303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:00.057701 24303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:00.067602 24308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:00.067643 24309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:00.067737 24303 server_base.cc:1034] running on GCE node
W20260620 21:50:00.067646 24311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:00.068835 24303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:00.070750 24303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:00.071974 24303 hybrid_clock.cc:648] HybridClock initialized: now 1781992200071934 us; error 57 us; skew 500 ppm
Jun 20 21:50:00 dist-test-slave-5bvn krb5kdc[24290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:00.076227 24303 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
I20260620 21:50:00.077310 24303 webserver.cc:460] Webserver started at http://127.1.155.190:46259/ using document root <none> and password file <none>
I20260620 21:50:00.078002 24303 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:00.078075 24303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:00.078306 24303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:00.080660 24303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/data/instance:
uuid: "7518d25c3b0d425bb418ab65c303671d"
format_stamp: "Formatted at 2026-06-20 21:50:00 on dist-test-slave-5bvn"
server_key: "85e9a8db80ba99feb6137ae22ae0eac6"
server_key_iv: "5fb446b75fd1924ead779a10aa74783f"
server_key_version: "encryptionkey@0"
I20260620 21:50:00.081305 24303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/wal/instance:
uuid: "7518d25c3b0d425bb418ab65c303671d"
format_stamp: "Formatted at 2026-06-20 21:50:00 on dist-test-slave-5bvn"
server_key: "85e9a8db80ba99feb6137ae22ae0eac6"
server_key_iv: "5fb446b75fd1924ead779a10aa74783f"
server_key_version: "encryptionkey@0"
I20260620 21:50:00.085517 24303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 21:50:00.088662 24318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:00.090226 24303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:00.090404 24303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/wal
uuid: "7518d25c3b0d425bb418ab65c303671d"
format_stamp: "Formatted at 2026-06-20 21:50:00 on dist-test-slave-5bvn"
server_key: "85e9a8db80ba99feb6137ae22ae0eac6"
server_key_iv: "5fb446b75fd1924ead779a10aa74783f"
server_key_version: "encryptionkey@0"
I20260620 21:50:00.090538 24303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:00.102730 24303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:00.106295 24303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:00.106521 24303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:00.114732 24303 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:36259
I20260620 21:50:00.115108 24370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.190:36259 every 8 connection(s)
I20260620 21:50:00.115775 24303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:00.118669 24371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:00.122507 24371 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:50:00.124236 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 24303
I20260620 21:50:00.124388 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-0/wal/instance
I20260620 21:50:00.124673 1646 external_mini_cluster.cc:1428] Setting key afc382f1aa90b3d49c3950c800cac0ec
WARNING: no policy specified for kudu/127.1.155.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.189@KRBTEST.COM" created.
Jun 20 21:50:00 dist-test-slave-5bvn krb5kdc[24290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:50:00.149298 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.124880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36259 (local address 127.0.0.1:57318)
0620 21:50:00.125440 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.125513 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:00.126701 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:00.127600 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:00.127605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:00.128179 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:00.128436 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.128451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.130375 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.130378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.131159 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.131169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.132304 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.132307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.132396 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.133094 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:00.133111 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:00.141173 (+ 8062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:00.144393 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.144401 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.144413 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.144811 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.146159 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.146164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.146166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.146444 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.147354 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:00.147361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:00.147520 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 21:50:00.147710 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 21:50:00.148190 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":114,"thread_start_us":117,"threads_started":1}
I20260620 21:50:00.149297 24374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.125614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.1.155.190:36259)
0620 21:50:00.125912 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.125922 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.125963 (+ 41us) server_negotiation.cc:407] Connection header received
0620 21:50:00.127064 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.127072 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.127347 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.127557 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.128608 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.130214 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.131295 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.132095 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.132296 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.141447 (+ 9151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:00.141457 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:00.141470 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:00.141535 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:00.144226 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.145235 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.145242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.145244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.146018 (+ 774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.146665 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.146668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.146670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.147229 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:00.147376 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:00.147790 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 21:50:00.148102 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
I20260620 21:50:00.156014 24371 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:50:00.156850 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.156548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:35009 (local address 127.0.0.1:46712)
0620 21:50:00.156693 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.156778 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 21:50:00.157089 24320 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.155.189:35009: connect: Connection refused (error 111)
I20260620 21:50:00.158034 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.157768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:35009 (local address 127.0.0.1:46718)
0620 21:50:00.157909 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.157961 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 21:50:00.158210 24371 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.189:35009: Network error: Client connection negotiation failed: client connection to 127.1.155.189:35009: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.189.keytab.
Entry for principal kudu/127.1.155.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.189.keytab.
I20260620 21:50:00.174202 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.189:35009
--webserver_interface=127.1.155.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.189.keytab
--principal=kudu/127.1.155.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:36259,127.1.155.189:35009,127.1.155.188:38835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:50:00.207486 24371 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } attempt: 1
I20260620 21:50:00.208477 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.208127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:35009 (local address 127.0.0.1:46720)
0620 21:50:00.208307 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.208381 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 21:50:00.209497 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.209209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:35009 (local address 127.0.0.1:46736)
0620 21:50:00.209350 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.209416 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 21:50:00.209695 24371 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.189:35009: Network error: Client connection negotiation failed: client connection to 127.1.155.189:35009: connect: Connection refused (error 111)
I20260620 21:50:00.277925 24371 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } attempt: 2
I20260620 21:50:00.278844 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.278486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:35009 (local address 127.0.0.1:46740)
0620 21:50:00.278686 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.278747 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 21:50:00.279678 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.279436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:35009 (local address 127.0.0.1:46754)
0620 21:50:00.279546 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.279620 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.189:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 21:50:00.279871 24371 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.189:35009: Network error: Client connection negotiation failed: client connection to 127.1.155.189:35009: connect: Connection refused (error 111)
W20260620 21:50:00.290043 24378 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:50:00.290414 24378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:00.290481 24378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:00.290575 24378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:00.294831 24378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:00.294921 24378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:00.294957 24378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:00.295015 24378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:00.295063 24378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:00.300631 24378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:36259,127.1.155.189:35009,127.1.155.188:38835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.189.keytab
--principal=kudu/127.1.155.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.189:35009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.155.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:00.301749 24378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:00.302789 24378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:00.311434 24384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:00.311587 24386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:00.311434 24383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:00.311805 24378 server_base.cc:1034] running on GCE node
I20260620 21:50:00.312494 24378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:00.313618 24378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:00.314829 24378 hybrid_clock.cc:648] HybridClock initialized: now 1781992200314825 us; error 51 us; skew 500 ppm
Jun 20 21:50:00 dist-test-slave-5bvn krb5kdc[24290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:00.317646 24378 init.cc:377] Logged in from keytab as kudu/127.1.155.189@KRBTEST.COM (short username kudu)
I20260620 21:50:00.318634 24378 webserver.cc:460] Webserver started at http://127.1.155.189:39347/ using document root <none> and password file <none>
I20260620 21:50:00.319190 24378 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:00.319278 24378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:00.319494 24378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:00.321514 24378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/data/instance:
uuid: "f2e1f85ca4694f55af119c2ed2812c97"
format_stamp: "Formatted at 2026-06-20 21:50:00 on dist-test-slave-5bvn"
server_key: "b3bfcc4671d9a7f8ecf2762eddb6a4e4"
server_key_iv: "b6faf59498935688eeb70ff01d392089"
server_key_version: "encryptionkey@0"
I20260620 21:50:00.322103 24378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/wal/instance:
uuid: "f2e1f85ca4694f55af119c2ed2812c97"
format_stamp: "Formatted at 2026-06-20 21:50:00 on dist-test-slave-5bvn"
server_key: "b3bfcc4671d9a7f8ecf2762eddb6a4e4"
server_key_iv: "b6faf59498935688eeb70ff01d392089"
server_key_version: "encryptionkey@0"
I20260620 21:50:00.325932 24378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 21:50:00.328461 24393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:00.329952 24378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 21:50:00.330096 24378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/wal
uuid: "f2e1f85ca4694f55af119c2ed2812c97"
format_stamp: "Formatted at 2026-06-20 21:50:00 on dist-test-slave-5bvn"
server_key: "b3bfcc4671d9a7f8ecf2762eddb6a4e4"
server_key_iv: "b6faf59498935688eeb70ff01d392089"
server_key_version: "encryptionkey@0"
I20260620 21:50:00.330226 24378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:00.344988 24378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:00.348065 24378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:00.348274 24378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:00.355425 24378 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.189:35009
I20260620 21:50:00.355768 24445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.189:35009 every 8 connection(s)
I20260620 21:50:00.356393 24378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/data/info.pb
I20260620 21:50:00.359102 24446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:00.361188 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 24378
I20260620 21:50:00.361325 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-1/wal/instance
I20260620 21:50:00.361611 1646 external_mini_cluster.cc:1428] Setting key 9995e66c5bf38dd2c6d85c04f79c8ece
I20260620 21:50:00.363690 24446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.155.188@KRBTEST.COM; defaulting to no policy
I20260620 21:50:00.371060 24371 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.155.188@KRBTEST.COM" created.
Jun 20 21:50:00 dist-test-slave-5bvn krb5kdc[24290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.189@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
Jun 20 21:50:00 dist-test-slave-5bvn krb5kdc[24290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for kudu/127.1.155.189@KRBTEST.COM
I20260620 21:50:00.381670 24374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.365535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.1.155.190:36259)
0620 21:50:00.365730 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.365737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.366628 (+ 891us) server_negotiation.cc:407] Connection header received
0620 21:50:00.367582 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.367587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.367651 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.367763 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.368976 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.369790 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.371344 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.371878 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.372315 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.376492 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:00.376496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:00.376499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:00.376527 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:00.378204 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.379022 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.379029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.379034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.379123 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.379577 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.379584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.379588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.379826 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:00.379940 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:00.381341 (+ 1401us) server_negotiation.cc:299] Negotiation successful
0620 21:50:00.381477 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:50:00.382212 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.371548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:35009 (local address 127.0.0.1:46760)
0620 21:50:00.371673 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.371690 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:00.371773 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:00.372634 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:00.372640 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:00.372837 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:00.372991 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.372999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.374410 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.374416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.374947 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.374954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.375547 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.375551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.375669 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.376165 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:00.376182 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:00.377763 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:00.380352 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.380356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.380358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.380569 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.380892 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.380897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.380899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.380981 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.381432 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:00.381435 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:00.381529 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 21:50:00.381650 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 21:50:00.381791 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 21:50:00.383029 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.371811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.1.155.189:35009)
0620 21:50:00.372171 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.372184 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.372206 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:50:00.372243 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.372248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.372451 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.372635 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.373108 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.374306 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.375064 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.375413 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.375530 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.377962 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:00.377971 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:00.377989 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:00.378016 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:00.380239 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.380700 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.380706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.380708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.380796 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.381075 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.381080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.381082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.381323 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:00.381464 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:00.381754 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 21:50:00.382115 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":182,"threads_started":1}
I20260620 21:50:00.383054 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.365993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36259 (local address 127.0.0.1:57324)
0620 21:50:00.366455 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.366509 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:00.367367 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:00.367938 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:00.367945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:00.368347 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:00.368814 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.368827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.370012 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.370016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.371061 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.371074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.372246 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.372249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.372441 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.374300 (+ 1859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:00.374327 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:00.376345 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:00.378330 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.378343 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.378359 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.378873 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.379250 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.379255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.379258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.379453 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.379964 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:00.379972 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:00.380343 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 21:50:00.381922 (+ 1579us) client_negotiation.cc:240] Negotiation successful
0620 21:50:00.382218 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":82,"thread_start_us":130,"threads_started":1}
I20260620 21:50:00.384310 24446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:50:00.385257 24371 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:50:00.386034 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.385758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:38835 (local address 127.0.0.1:50130)
0620 21:50:00.385934 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.385979 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 21:50:00.386772 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.386525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:38835 (local address 127.0.0.1:50146)
0620 21:50:00.386669 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.386717 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 21:50:00.386933 24371 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:38835: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
Jun 20 21:50:00 dist-test-slave-5bvn krb5kdc[24290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.189@KRBTEST.COM for kudu/127.1.155.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.155.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.188.keytab.
Entry for principal kudu/127.1.155.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.188.keytab.
I20260620 21:50:00.391189 1646 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeoPTss/build/debug/bin/kudu
/tmp/dist-test-taskeoPTss/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.188:38835
--webserver_interface=127.1.155.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.188.keytab
--principal=kudu/127.1.155.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.155.148:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.155.190:36259,127.1.155.189:35009,127.1.155.188:38835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:50:00.393410 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.384928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46764 (local address 127.1.155.189:35009)
0620 21:50:00.385023 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.385027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.385040 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:50:00.385106 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.385109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.385156 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.385261 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.385690 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.386482 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.387370 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.387713 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.387808 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.389926 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:00.389929 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:00.389932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:00.389976 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:00.391744 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.392190 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.392197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.392202 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.392292 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.392573 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.392587 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.392588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.392758 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:00.392859 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:00.393084 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 21:50:00.393220 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 21:50:00.393424 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.384863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:35009 (local address 127.0.0.1:46764)
0620 21:50:00.384956 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.384969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:00.385044 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:00.385269 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:00.385274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:00.385441 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:00.385555 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.385565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.386587 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.386592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.387261 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.387270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.387845 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.387848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.387927 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.388397 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:00.388408 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:00.389742 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:00.391871 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.391873 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.391875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.392062 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.392388 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.392390 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.392392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.392466 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.392859 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:00.392861 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:00.392938 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 21:50:00.393042 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 21:50:00.393279 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":83}
I20260620 21:50:00.394197 24446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:50:00.394912 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.394664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:38835 (local address 127.0.0.1:50162)
0620 21:50:00.394775 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.394857 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 21:50:00.395509 24394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
I20260620 21:50:00.396340 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.396133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:38835 (local address 127.0.0.1:50176)
0620 21:50:00.396239 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.396282 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 21:50:00.396493 24446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:38835: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
I20260620 21:50:00.418128 24371 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } attempt: 1
I20260620 21:50:00.419006 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.418720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:38835 (local address 127.0.0.1:50190)
0620 21:50:00.418872 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.418933 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 21:50:00.419739 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.419527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:38835 (local address 127.0.0.1:50204)
0620 21:50:00.419636 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.419687 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 21:50:00.419870 24371 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:38835: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
I20260620 21:50:00.445739 24446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } attempt: 1
I20260620 21:50:00.446615 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.446331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:38835 (local address 127.0.0.1:50206)
0620 21:50:00.446471 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.446539 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 21:50:00.447353 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.447127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:38835 (local address 127.0.0.1:50208)
0620 21:50:00.447237 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.447289 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 21:50:00.447490 24446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:38835: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
I20260620 21:50:00.495050 24371 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } attempt: 2
I20260620 21:50:00.495852 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.495570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:38835 (local address 127.0.0.1:50210)
0620 21:50:00.495719 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.495784 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 21:50:00.496549 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.496343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:38835 (local address 127.0.0.1:50224)
0620 21:50:00.496447 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.496501 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 21:50:00.496662 24371 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:38835: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
W20260620 21:50:00.504966 24452 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:50:00.505329 24452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:00.505393 24452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:00.505475 24452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:00.509701 24452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:00.509799 24452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:00.509833 24452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:00.509915 24452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:00.509953 24452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:00.515338 24452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.155.148:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.155.190:36259,127.1.155.189:35009,127.1.155.188:38835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/krb5kdc/kudu_127.1.155.188.keytab
--principal=kudu/127.1.155.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.155.188:38835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.155.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:00.515690 24446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } attempt: 2
I20260620 21:50:00.516501 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.516204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:38835 (local address 127.0.0.1:50240)
0620 21:50:00.516349 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.516426 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 21:50:00.516570 24452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:00.517225 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.517003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:38835 (local address 127.0.0.1:50242)
0620 21:50:00.517117 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.517168 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 21:50:00.517436 24446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.155.188:38835: Network error: Client connection negotiation failed: client connection to 127.1.155.188:38835: connect: Connection refused (error 111)
I20260620 21:50:00.517711 24452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:00.526093 24458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:00.526108 24461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:00.526389 24452 server_base.cc:1034] running on GCE node
W20260620 21:50:00.526108 24459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:00.526994 24452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:00.528237 24452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:00.529412 24452 hybrid_clock.cc:648] HybridClock initialized: now 1781992200529397 us; error 28 us; skew 500 ppm
Jun 20 21:50:00 dist-test-slave-5bvn krb5kdc[24290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:00.532299 24452 init.cc:377] Logged in from keytab as kudu/127.1.155.188@KRBTEST.COM (short username kudu)
I20260620 21:50:00.533249 24452 webserver.cc:460] Webserver started at http://127.1.155.188:43801/ using document root <none> and password file <none>
I20260620 21:50:00.533926 24452 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:00.534026 24452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:00.534255 24452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:00.536332 24452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/data/instance:
uuid: "04117de34e76499e829b75e3926c1d97"
format_stamp: "Formatted at 2026-06-20 21:50:00 on dist-test-slave-5bvn"
server_key: "2a053141064a7b0d10e74a984ed5cb09"
server_key_iv: "bbac395a0d734f6fe11b30c3d26a5151"
server_key_version: "encryptionkey@0"
I20260620 21:50:00.536978 24452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/wal/instance:
uuid: "04117de34e76499e829b75e3926c1d97"
format_stamp: "Formatted at 2026-06-20 21:50:00 on dist-test-slave-5bvn"
server_key: "2a053141064a7b0d10e74a984ed5cb09"
server_key_iv: "bbac395a0d734f6fe11b30c3d26a5151"
server_key_version: "encryptionkey@0"
I20260620 21:50:00.541138 24452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 21:50:00.543860 24468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:00.545477 24452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 21:50:00.545656 24452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/data,/tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/wal
uuid: "04117de34e76499e829b75e3926c1d97"
format_stamp: "Formatted at 2026-06-20 21:50:00 on dist-test-slave-5bvn"
server_key: "2a053141064a7b0d10e74a984ed5cb09"
server_key_iv: "bbac395a0d734f6fe11b30c3d26a5151"
server_key_version: "encryptionkey@0"
I20260620 21:50:00.545792 24452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:00.556543 24452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:00.559798 24452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:00.560017 24452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:00.567217 24452 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.188:38835
I20260620 21:50:00.567549 24520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.155.188:38835 every 8 connection(s)
I20260620 21:50:00.568221 24452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/data/info.pb
I20260620 21:50:00.568523 1646 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeoPTss/build/debug/bin/kudu as pid 24452
I20260620 21:50:00.568686 1646 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeoPTss/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781991920067886-1646-0/minicluster-data/master-2/wal/instance
I20260620 21:50:00.569000 1646 external_mini_cluster.cc:1428] Setting key 002f1b6b2c6051273acd60b264ffe123
I20260620 21:50:00.571394 24521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:00.575911 24521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:50:00 dist-test-slave-5bvn krb5kdc[24290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.188@KRBTEST.COM
Jun 20 21:50:00 dist-test-slave-5bvn krb5kdc[24290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.188@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
I20260620 21:50:00.584158 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.570307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50248 (local address 127.1.155.188:38835)
0620 21:50:00.570683 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.570709 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.570743 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:50:00.571376 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.571391 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.571767 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.572104 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.573113 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.574225 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.575419 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.575890 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.576008 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.579231 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:00.579236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:00.579247 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:00.579274 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:00.580970 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.581418 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.581423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.581427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.581539 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.581816 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.581821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.581823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.582391 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:00.582605 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:00.582841 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 21:50:00.583135 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":134,"threads_started":1}
I20260620 21:50:00.587594 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.578280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.190:36259 (local address 127.0.0.1:57334)
0620 21:50:00.578750 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.578791 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:00.578935 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:00.579231 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:00.579236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:00.579616 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:00.579743 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.579749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.580503 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.580506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.581125 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.581134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.581596 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.581600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.581727 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.582289 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:00.582303 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:00.583869 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:00.585680 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.585691 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.585697 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.586011 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.586358 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.586361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.586364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.586472 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.586985 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:00.586991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:00.587120 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 21:50:00.587276 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 21:50:00.587431 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":163,"threads_started":1}
I20260620 21:50:00.587708 24374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.578303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57334 (local address 127.1.155.190:36259)
0620 21:50:00.578452 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.578456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.578874 (+ 418us) server_negotiation.cc:407] Connection header received
0620 21:50:00.579056 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.579059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.579112 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.579200 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.579867 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.580375 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.581250 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.581482 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.581593 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.584012 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:00.584015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:00.584017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:00.584040 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:00.585454 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.586115 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.586119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.586121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.586233 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.586630 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.586634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.586636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.586874 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:00.586962 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:00.587371 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 21:50:00.587504 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:50:00.588798 24521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:50:00 dist-test-slave-5bvn krb5kdc[24290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
Jun 20 21:50:00 dist-test-slave-5bvn krb5kdc[24290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.188@KRBTEST.COM for kudu/127.1.155.189@KRBTEST.COM
I20260620 21:50:00.595832 24371 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } attempt: 3
I20260620 21:50:00.597131 24526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.586548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.1.155.190:36259)
0620 21:50:00.586908 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.586917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.586950 (+ 33us) server_negotiation.cc:407] Connection header received
0620 21:50:00.587026 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.587029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.587074 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.587175 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.588141 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.588800 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.589949 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.590189 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.590300 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.592888 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:00.592893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:00.592896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:00.592922 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:00.594675 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.595297 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.595303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.595306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.595405 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.595757 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.595760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.595761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.596075 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:00.596294 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:00.596763 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 21:50:00.596905 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":123,"threads_started":1}
I20260620 21:50:00.597985 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.589326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.189:35009 (local address 127.0.0.1:46778)
0620 21:50:00.589469 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.589482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:00.589546 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:00.589766 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:00.589768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:00.590007 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:00.590102 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.590107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.590966 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.590979 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.591497 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.591504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.592105 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.592108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.592186 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.592691 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:00.592703 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:00.594220 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:00.596229 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.596232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.596234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.596471 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.596934 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.596938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.596942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.597031 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.597478 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:00.597481 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:00.597568 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 21:50:00.597668 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 21:50:00.597789 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 21:50:00.598482 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.589435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.1.155.189:35009)
0620 21:50:00.589549 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.589554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.589594 (+ 40us) server_negotiation.cc:407] Connection header received
0620 21:50:00.589630 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.589632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.589670 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.589728 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.590221 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.590847 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.591617 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.591955 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.592062 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.594364 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:00.594366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:00.594368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:00.594393 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:00.596079 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.596742 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.596745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.596746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.596837 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.597125 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.597129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.597131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.597346 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:00.597431 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:00.598263 (+ 832us) server_negotiation.cc:299] Negotiation successful
0620 21:50:00.598376 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 21:50:00.599210 24521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:50:00 dist-test-slave-5bvn krb5kdc[24290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for kudu/127.1.155.188@KRBTEST.COM
Jun 20 21:50:00 dist-test-slave-5bvn krb5kdc[24290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.189@KRBTEST.COM
Jun 20 21:50:00 dist-test-slave-5bvn krb5kdc[24290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.188@KRBTEST.COM for kudu/127.1.155.188@KRBTEST.COM
I20260620 21:50:00.605762 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.596306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:38835 (local address 127.0.0.1:50264)
0620 21:50:00.596424 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.596438 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:00.596504 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:00.596906 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:00.596909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:00.597062 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:00.597153 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.597161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.598563 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.598566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.599287 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.599296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.600015 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.600019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.600120 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.600607 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:00.600619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:00.601984 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:00.604046 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.604051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.604054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.604314 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.604621 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.604624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.604626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.604722 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.605145 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:00.605151 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:00.605256 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 21:50:00.605390 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 21:50:00.605528 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 21:50:00.606417 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.596558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50264 (local address 127.1.155.188:38835)
0620 21:50:00.596697 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.596702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.596715 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:50:00.596744 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.596747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.596792 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.596871 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.597288 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.598446 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.599411 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.599787 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.599878 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.602119 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:00.602122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:00.602124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:00.602147 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:00.603904 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.604415 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.604420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.604425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.604500 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.604799 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.604803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.604805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.604981 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:00.605097 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:00.605443 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 21:50:00.606215 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:50:00.608610 24446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } attempt: 3
I20260620 21:50:00.608780 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.598423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46788 (local address 127.1.155.189:35009)
0620 21:50:00.598639 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.598643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.599027 (+ 384us) server_negotiation.cc:407] Connection header received
0620 21:50:00.599158 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.599161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.599213 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.599303 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.600134 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.600787 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.601531 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.601913 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.602030 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.604752 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:00.604758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:00.604762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:00.604795 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:00.606550 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.606987 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.606992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.606994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.607085 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.607365 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.607368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.607370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.607585 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:00.607798 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:00.608426 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 21:50:00.608536 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":110,"threads_started":1}
I20260620 21:50:00.609695 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.599865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:38835 (local address 127.0.0.1:50274)
0620 21:50:00.599960 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.599974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:00.600050 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:00.600520 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:00.600523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:00.601185 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:00.601270 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.601277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.602288 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.602293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.603026 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.603040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.603690 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.603695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.603788 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.604284 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:00.604299 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:00.606092 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:00.607940 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.607945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.607949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.608223 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.608503 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.608506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.608507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.608606 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.609214 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:00.609216 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:00.609300 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 21:50:00.609401 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 21:50:00.609539 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 21:50:00.610138 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.599989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50274 (local address 127.1.155.188:38835)
0620 21:50:00.600297 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.600306 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.600318 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:50:00.600348 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.600351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.600392 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.600497 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.601395 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.602187 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.603145 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.603544 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.603678 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.606213 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:00.606216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:00.606218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:00.606237 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:00.607838 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.608314 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.608320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.608323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.608401 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.608701 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.608704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.608706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.609086 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:00.609180 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:00.609908 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 21:50:00.610013 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":86,"threads_started":1}
I20260620 21:50:00.610649 24371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d: Bootstrap starting.
I20260620 21:50:00.612870 24521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97: Bootstrap starting.
I20260620 21:50:00.613435 24371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:00.614413 24371 log.cc:824] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:50:00 dist-test-slave-5bvn krb5kdc[24290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.189@KRBTEST.COM for kudu/127.1.155.188@KRBTEST.COM
I20260620 21:50:00.615411 24521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:00.616250 24521 log.cc:824] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:00.617559 24371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d: No bootstrap required, opened a new log
I20260620 21:50:00.617712 1646 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:50:00.618255 24521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97: No bootstrap required, opened a new log
I20260620 21:50:00.618940 24345 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7518d25c3b0d425bb418ab65c303671d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57342
I20260620 21:50:00.620543 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.609003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.155.188:38835 (local address 127.0.0.1:50284)
0620 21:50:00.609139 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:00.609156 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:00.609233 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:00.609577 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:00.609580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:00.609741 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:00.610083 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.610096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.611334 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.611337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.612110 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:00.612121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.612691 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.612694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:00.612775 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.613317 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:00.613334 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:00.615017 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:00.618816 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.618820 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.618822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.619088 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.619397 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:00.619401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:00.619403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:00.619487 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:00.619923 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:00.619929 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:00.620028 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 21:50:00.620129 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 21:50:00.620377 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 21:50:00.621294 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.609267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50284 (local address 127.1.155.188:38835)
0620 21:50:00.609371 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.609377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.609390 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:50:00.609423 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.609425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.609461 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.609536 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.610211 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.611198 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.612258 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.612564 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.612674 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.615160 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:00.615164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:00.615167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:00.615190 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:00.618687 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.619213 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.619216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.619218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.619301 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.619594 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.619597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.619598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.619805 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:00.620929 (+ 1124us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:00.621004 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 21:50:00.621131 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 21:50:00.622063 24371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } }
I20260620 21:50:00.622092 24521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } }
I20260620 21:50:00.622292 24371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7518d25c3b0d425bb418ab65c303671d, State: Initialized, Role: FOLLOWER
I20260620 21:50:00.622310 24521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04117de34e76499e829b75e3926c1d97, State: Initialized, Role: FOLLOWER
I20260620 21:50:00.622627 24371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } }
I20260620 21:50:00.622635 24521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } }
I20260620 21:50:00.623273 24532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } } }
I20260620 21:50:00.623517 24532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:50:00.624109 24521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:00.624109 24371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:00.624143 24345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 21:50:00.624243 24345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:00.624583 24446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97: Bootstrap starting.
I20260620 21:50:00.625582 24345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } }
I20260620 21:50:00.626610 24345 leader_election.cc:288] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [CANDIDATE]: Term 1 election: Requested vote from peers f2e1f85ca4694f55af119c2ed2812c97 (127.1.155.189:35009), 04117de34e76499e829b75e3926c1d97 (127.1.155.188:38835)
I20260620 21:50:00.628592 24495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7518d25c3b0d425bb418ab65c303671d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "04117de34e76499e829b75e3926c1d97"
I20260620 21:50:00.629465 24495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97 [term 0 FOLLOWER]: Advancing to term 1
W20260620 21:50:00.630141 24553 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:50:00.630211 24553 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:50:00.630970 24495 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7518d25c3b0d425bb418ab65c303671d in term 1.
I20260620 21:50:00.631400 24319 leader_election.cc:302] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04117de34e76499e829b75e3926c1d97, 7518d25c3b0d425bb418ab65c303671d; no voters:
I20260620 21:50:00.631919 24561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } } }
I20260620 21:50:00.632138 24561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:50:00.632177 24446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:00.632311 24561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:00.632463 24561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [term 1 LEADER]: Becoming Leader. State: Replica: 7518d25c3b0d425bb418ab65c303671d, State: Running, Role: LEADER
I20260620 21:50:00.633013 24561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } }
I20260620 21:50:00.633117 24446 log.cc:824] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:00.632409 24303 master_runner.cc:186] Error getting master registration for 127.1.155.189:35009: OK, instance_id { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" instance_seqno: 1781992200353365 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:50:00.635860 24303 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:50:00.633147 24452 master_runner.cc:186] Error getting master registration for 127.1.155.189:35009: OK, instance_id { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" instance_seqno: 1781992200353365 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:50:00.636430 24452 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:50:00.636539 24561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7518d25c3b0d425bb418ab65c303671d. Latest consensus state: current_term: 1 leader_uuid: "7518d25c3b0d425bb418ab65c303671d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } } }
I20260620 21:50:00.636744 24561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [sys.catalog]: This master's current role is: LEADER
W20260620 21:50:00.636765 24554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:50:00.637130 24554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:50:00.637780 24565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:00.638862 24565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:00.640412 24446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97: No bootstrap required, opened a new log
I20260620 21:50:00.647294 24446 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } }
I20260620 21:50:00.647703 24446 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e1f85ca4694f55af119c2ed2812c97, State: Initialized, Role: FOLLOWER
I20260620 21:50:00.648216 24446 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } }
I20260620 21:50:00.650656 24420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7518d25c3b0d425bb418ab65c303671d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f2e1f85ca4694f55af119c2ed2812c97"
I20260620 21:50:00.650908 24420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:00.652361 24420 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7518d25c3b0d425bb418ab65c303671d in term 1.
I20260620 21:50:00.652599 24446 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:00.654609 24526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.633936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.1.155.190:36259)
0620 21:50:00.634060 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.634064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.642178 (+ 8114us) server_negotiation.cc:407] Connection header received
0620 21:50:00.642383 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.642387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.642447 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.642504 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.643632 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.644656 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.645968 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.646361 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.646449 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.649383 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:00.649387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:00.649390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:00.649424 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:00.651326 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.651892 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.651895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.651897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.652237 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.653597 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.653601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.653603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.653820 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:00.653946 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:00.654317 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 21:50:00.654445 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:50:00.661604 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.638269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.1.155.189:35009)
0620 21:50:00.638397 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.638402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.646131 (+ 7729us) server_negotiation.cc:407] Connection header received
0620 21:50:00.646318 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.646322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.646377 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.646429 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.647432 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.648162 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.651114 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.651424 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.651502 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.656345 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:00.656349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:00.656354 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:00.656384 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:00.658300 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.659363 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.659369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.659372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.659532 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.660049 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.660053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.660055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.660325 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:00.660450 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:00.661332 (+ 882us) server_negotiation.cc:299] Negotiation successful
0620 21:50:00.661477 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:50:00.666317 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:00.639034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50296 (local address 127.1.155.188:38835)
0620 21:50:00.639157 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.639164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.648831 (+ 9667us) server_negotiation.cc:407] Connection header received
0620 21:50:00.649343 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.649346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.649405 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.649460 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.651986 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.652698 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.654344 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:00.654679 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:00.654770 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:00.662768 (+ 7998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:00.662776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:00.662778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:00.662806 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:00.664547 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.664991 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.664994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.664996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.665081 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:00.665362 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:00.665366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:00.665367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:00.665557 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:00.665725 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:00.666036 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 21:50:00.666151 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:50:00.666827 24569 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } } }
I20260620 21:50:00.667775 24569 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:50:00.668334 24420 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97 [term 1 FOLLOWER]: Refusing update from remote peer 7518d25c3b0d425bb418ab65c303671d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:00.668372 24495 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97 [term 1 FOLLOWER]: Refusing update from remote peer 7518d25c3b0d425bb418ab65c303671d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:00.668996 24561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [LEADER]: Connected to new peer: Peer: permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:00.669466 24561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:00.673137 24532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7518d25c3b0d425bb418ab65c303671d. Latest consensus state: current_term: 1 leader_uuid: "7518d25c3b0d425bb418ab65c303671d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } } }
I20260620 21:50:00.673286 24532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:50:00.675187 24569 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7518d25c3b0d425bb418ab65c303671d. Latest consensus state: current_term: 1 leader_uuid: "7518d25c3b0d425bb418ab65c303671d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } } }
I20260620 21:50:00.675323 24569 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 21:50:00.678447 24580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:50:00.678545 24580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:50:00.684376 24531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7518d25c3b0d425bb418ab65c303671d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } } }
I20260620 21:50:00.684540 24531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:00.685891 24532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7518d25c3b0d425bb418ab65c303671d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } } }
I20260620 21:50:00.686023 24532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:50:00.686690 24582 mvcc.cc:205] Tried to move back new op lower bound from 7299040053930176512 to 7299040053800804352. Current Snapshot: MvccSnapshot[applied={T|T < 7299040053930176512}]
I20260620 21:50:00.693437 24565 catalog_manager.cc:1347] Generated new cluster ID: c87058b40b5b4e2399bf16d7bd292b69
I20260620 21:50:00.693535 24565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:00.697747 24561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7518d25c3b0d425bb418ab65c303671d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } } }
I20260620 21:50:00.697914 24561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:00.699471 24569 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7518d25c3b0d425bb418ab65c303671d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7518d25c3b0d425bb418ab65c303671d" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 36259 } } peers { permanent_uuid: "f2e1f85ca4694f55af119c2ed2812c97" member_type: VOTER last_known_addr { host: "127.1.155.189" port: 35009 } } peers { permanent_uuid: "04117de34e76499e829b75e3926c1d97" member_type: VOTER last_known_addr { host: "127.1.155.188" port: 38835 } } }
I20260620 21:50:00.699743 24569 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:50:00.703614 24581 mvcc.cc:205] Tried to move back new op lower bound from 7299040053930176512 to 7299040053800804352. Current Snapshot: MvccSnapshot[applied={T|T < 7299040053930176512}]
I20260620 21:50:00.713909 24565 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:00.715224 24565 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:00.725730 24565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7518d25c3b0d425bb418ab65c303671d: Generated new TSK 0
I20260620 21:50:00.726645 24565 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:50:00.766978 24450 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:00.763992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.1.155.189:35009)
0620 21:50:00.764122 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.764126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.765698 (+ 1572us) server_negotiation.cc:407] Connection header received
0620 21:50:00.765956 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.765960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.766014 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.766069 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.766833 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46812: BlockingRecv error: recv got EOF from 127.0.0.1:46812 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 21:50:00.768421 24524 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:00.763529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50304 (local address 127.1.155.188:38835)
0620 21:50:00.763667 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.763672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.767495 (+ 3823us) server_negotiation.cc:407] Connection header received
0620 21:50:00.767675 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.767678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.767732 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.767784 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.768299 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50304: BlockingRecv error: recv got EOF from 127.0.0.1:50304 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 21:50:00.769767 24526 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:00.764471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.1.155.190:36259)
0620 21:50:00.764585 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.764589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.768885 (+ 4296us) server_negotiation.cc:407] Connection header received
0620 21:50:00.769057 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.769060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.769126 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.769178 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.769656 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57358: BlockingRecv error: recv got EOF from 127.0.0.1:57358 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 21:50:00.774737 24450 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:00.773026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.1.155.189:35009)
0620 21:50:00.773153 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.773157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.773706 (+ 549us) server_negotiation.cc:407] Connection header received
0620 21:50:00.773974 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.773977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.774056 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.774123 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.774630 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46824: BlockingRecv error: recv got EOF from 127.0.0.1:46824 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 21:50:00.780187 24450 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:00.778575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46830 (local address 127.1.155.189:35009)
0620 21:50:00.778686 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.778690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.779245 (+ 555us) server_negotiation.cc:407] Connection header received
0620 21:50:00.779431 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.779437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.779488 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.779541 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.780058 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46830: BlockingRecv error: recv got EOF from 127.0.0.1:46830 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 21:50:00.786185 24450 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:00.784530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46846 (local address 127.1.155.189:35009)
0620 21:50:00.784640 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.784644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.785209 (+ 565us) server_negotiation.cc:407] Connection header received
0620 21:50:00.785394 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.785398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.785450 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.785501 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.786060 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46846: BlockingRecv error: recv got EOF from 127.0.0.1:46846 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 21:50:00.794481 24450 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:00.792615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.1.155.189:35009)
0620 21:50:00.792773 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.792777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.793403 (+ 626us) server_negotiation.cc:407] Connection header received
0620 21:50:00.793604 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.793608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.793666 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.793725 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.794365 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46860: BlockingRecv error: recv got EOF from 127.0.0.1:46860 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260620 21:50:00.807107 24450 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:00.805187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46866 (local address 127.1.155.189:35009)
0620 21:50:00.805322 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.805327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.805998 (+ 671us) server_negotiation.cc:407] Connection header received
0620 21:50:00.806252 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.806256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.806316 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.806375 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.806968 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46866: BlockingRecv error: recv got EOF from 127.0.0.1:46866 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 21:50:00.827605 24450 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:00.825989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.1.155.189:35009)
0620 21:50:00.826111 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.826115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.826656 (+ 541us) server_negotiation.cc:407] Connection header received
0620 21:50:00.826845 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.826849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.826900 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.826969 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.827494 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46876: BlockingRecv error: recv got EOF from 127.0.0.1:46876 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 21:50:00.863451 24450 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:00.862190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.1.155.189:35009)
0620 21:50:00.862318 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.862323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.862571 (+ 248us) server_negotiation.cc:407] Connection header received
0620 21:50:00.862734 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.862738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.862794 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.862884 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.863332 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46888: BlockingRecv error: recv got EOF from 127.0.0.1:46888 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 21:50:00.930886 24450 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:00.929615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46902 (local address 127.1.155.189:35009)
0620 21:50:00.929776 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:00.929782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:00.930016 (+ 234us) server_negotiation.cc:407] Connection header received
0620 21:50:00.930197 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:00.930200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:00.930247 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:00.930341 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:00.930784 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46902: BlockingRecv error: recv got EOF from 127.0.0.1:46902 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 21:50:01.062331 24450 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:01.061046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46908 (local address 127.1.155.189:35009)
0620 21:50:01.061235 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:01.061240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:01.061374 (+ 134us) server_negotiation.cc:407] Connection header received
0620 21:50:01.061548 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:01.061552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:01.061613 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:01.061708 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:01.062193 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46908: BlockingRecv error: recv got EOF from 127.0.0.1:46908 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 21:50:01.321473 24450 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:01.320543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.1.155.189:35009)
0620 21:50:01.320699 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:01.320702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:01.320718 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:50:01.320847 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:01.320849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:01.320898 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:01.320983 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:01.321379 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46916: BlockingRecv error: recv got EOF from 127.0.0.1:46916 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 21:50:01.631467 24553 catalog_manager.cc:1259] Loaded cluster ID: c87058b40b5b4e2399bf16d7bd292b69
I20260620 21:50:01.631609 24553 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97: loading cluster ID for follower catalog manager: success
I20260620 21:50:01.633298 24553 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97: acquiring CA information for follower catalog manager: success
I20260620 21:50:01.634613 24553 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04117de34e76499e829b75e3926c1d97: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:50:01.679941 24580 catalog_manager.cc:1259] Loaded cluster ID: c87058b40b5b4e2399bf16d7bd292b69
I20260620 21:50:01.680037 24580 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97: loading cluster ID for follower catalog manager: success
I20260620 21:50:01.681792 24580 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97: acquiring CA information for follower catalog manager: success
I20260620 21:50:01.683030 24580 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2e1f85ca4694f55af119c2ed2812c97: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:50:01.840020 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:01.835763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.1.155.189:35009)
0620 21:50:01.835995 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:01.836000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:01.836145 (+ 145us) server_negotiation.cc:407] Connection header received
0620 21:50:01.836320 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:01.836324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:01.836454 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:01.836603 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:01.836856 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:01.837384 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:01.838333 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:01.838577 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:01.838690 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:01.839007 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:01.839569 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:01.839736 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:01.839773 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 21:50:01.839830 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":67,"threads_started":1}
I20260620 21:50:01.846957 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:01.843163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50306 (local address 127.1.155.188:38835)
0620 21:50:01.843410 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:01.843415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:01.843430 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:50:01.843610 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:01.843613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:01.843732 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:01.843857 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:01.844072 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:01.844582 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:01.845429 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:01.845670 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:01.845780 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:01.846103 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:01.846539 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:01.846675 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:01.846737 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 21:50:01.846795 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
I20260620 21:50:01.854465 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:01.850407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.1.155.188:38835)
0620 21:50:01.850514 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:01.850519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:01.850957 (+ 438us) server_negotiation.cc:407] Connection header received
0620 21:50:01.851101 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:01.851106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:01.851167 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:01.851238 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:01.851575 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:01.852169 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:01.853281 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:01.853554 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:01.853699 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:01.853986 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:01.854098 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:01.854178 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:01.854256 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 21:50:01.854311 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 21:50:01.855388 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:01.850697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.1.155.189:35009)
0620 21:50:01.850823 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:01.850828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:01.850843 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:50:01.851021 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:01.851024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:01.851072 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:01.851159 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:01.851404 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:01.852234 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:01.854140 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:01.854466 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:01.854543 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:01.854967 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:01.855050 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:01.855121 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:01.855217 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 21:50:01.855271 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:50:01.855958 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:01.850255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.1.155.190:36259)
0620 21:50:01.850830 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:01.850835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:01.850848 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:50:01.850947 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:01.850952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:01.851117 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:01.851226 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:01.851620 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:01.852409 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:01.854133 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:01.854495 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:01.854709 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:01.855131 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:01.855522 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:01.855616 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:01.855757 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 21:50:01.855821 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":77,"threads_started":1}
I20260620 21:50:01.859668 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 24303
I20260620 21:50:01.867231 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 24378
I20260620 21:50:01.873804 1646 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeoPTss/build/debug/bin/kudu with pid 24452
2026-06-20T21:50:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3123 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6266 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (281798 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST